Skip to content

Best Practice for uploading data to BigQuery in Java? #2424

Description

@Electricks94

Hello,

I want to upload data to an existing table in BigQuery. Therefore I looked in the docks and found bigquery.insertAll which works fine but it takes very long to execute my code. I think it´s because I upload my data row by row and every row takes about a second. In the docs I read about 10.000 rows per second but I don`t see a way to change my code in a way so that it can do so.

Is there a smart and especially fast way to upload data in BigQuery. I considered to get my data into Arraylists or Json Objects and upload it in one step but I couldn`t find a way which would do so.

Here is my Code to check what I did:

	public static void uploaddata(String datasetname) throws IOException {
		BigQuery bigquery = BigQueryOptions
				.getDefaultInstance()
				.toBuilder()
				.setProjectId("testprojekt-175408")
				.build()
				.getService();
		TableId tableIdor = TableId.of(datasetname, "review_test");
		String csvFile = "C:/Users/Marku/Desktop/testfile2.csv";
		BufferedReader br = null;
		FileReader myFile = null;
		String line = "";
		String cvsSplitBy = ";";
		String[] beschriftung = null;
		int i = 0;
		Map<String, Object> rowContent = new HashMap<>();
		myFile = new FileReader(csvFile);
		br = new BufferedReader(myFile);
		// read CSV file line by line and upload it into BigQuery
		while ((line = br.readLine()) != null) {
			// get the name of the fields from the first row of the CSV File
			if (i == 0) {
				beschriftung = line.split(cvsSplitBy);
				i = i + 1;
				for (int e = 0; e < beschriftung.length; e++) {
					rowContent.put(beschriftung[e], "init");
				}
			} else
			// Format Data for BigQuery and upload the row
			{
				String[] Zeile = line.split(cvsSplitBy);
				for (int e = 0; e < Zeile.length; e++) {
					rowContent.put(beschriftung[e], Zeile[e]);
				}
				i = i + 1;
			}
			InsertAllResponse response = bigquery
							.insertAll(InsertAllRequest
							.newBuilder(tableIdor)
							.addRow(String.valueOf(i), rowContent)
							.build());
			if (response.hasErrors()) {
				System.out.println(response.getErrorsFor(0));
			}
		}
		br.close();
		myFile.close();
	}
}

As you can see I get my data from a CSV file. There are several reasons why I don’t upload just the CSV with a load job:

  1. As I´m from Germany we have quite different Date and Number formatting. I want to do the formatting BigQuery requires line by line, so that nobody who wants to upload has to think about other formatting than normal here.
  2. Sometimes I need to add the data in the CSV to an existing table and I have no idea if it is possible to add a whole CSV file to an existing table.

thanks in advance for your help.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

api: bigqueryIssues related to the BigQuery API.priority: p2Moderately-important priority. Fix may not be included in next release.type: questionRequest for information or clarification. Not an issue.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions