Loading data from Firestore exports  |  BigQuery  |  Google Cloud (2023)

Console

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project, and then select a dataset.
3. In the Dataset info section, click add_box Create table.
4. In the Create table panel, specify the following details:
1. In the Source section, select Google Cloud Storage in the Create table from list. Then, do the following:
   1. Select a file from the Cloud Storage bucket, or enter the Cloud Storage URI. You cannot include multiple URIs in the Google Cloud console, but wildcards are supported. The Cloud Storage bucket must be in the same location as the dataset that contains the table you want to create, append, or overwrite.
      The URI for your Firestore export file must end with KIND_COLLECTION_ID.export_metadata. For example, in default_namespace_kind_Book.export_metadata, Book is the collection ID, and default_namespace_kind_Book is the file name generated by Firestore. If the URI doesn't end with KIND_COLLECTION_ID.export_metadata, you receive the following error message: does not contain valid backup metadata. (error code: invalid).
   2. For File format, select Cloud Datastore Backup. Firestore and Datastore share the export format.
2. In the Destination section, specify the following details:
   1. For Dataset, select the dataset in which you want to create the table.
   2. In the Table field, enter the name of the table that you want to create.
   3. Verify that the Table type field is set to Native table.
3. In the Schema section, no action is necessary. The schema is inferred for a Firestore export.
4. Optional: Specify Partition and cluster settings. For more information, see Creating partitioned tables and Creating and using clustered tables.
5. Click Advanced options and do the following:
   • For Write preference, leave Write if empty selected. This option creates a new table and loads your data into it.
   • If you want to ignore values in a row that are not present in the table's schema, then select Unknown values.
   • For Encryption, click Customer-managed key to use a Cloud Key Management Service key. If you leave the Google-managed key setting, BigQuery encrypts the data at rest.
6. Click Create table.

bq

Use the bq loadcommand with source_format set to DATASTORE_BACKUP.Supply the --location flag and set the value to yourlocation. If you are overwitingan existing table, add the --replace flag.

To load only specific fields, use the --projection_fields flag.

bq --location=LOCATION load \--source_format=FORMAT \DATASET.TABLE \PATH_TO_SOURCE

Replace the following:

• LOCATION: your location. The --location flagis optional.
• FORMAT: DATASTORE_BACKUP.Datastore Backup is the correct option for Firestore.Firestore and Datastore share an export format.
• DATASET: the dataset that contains the tableinto which you're loading data.
• TABLE: the table into which you're loading data.If the table doesn't exist, it is created.
• PATH_TO_SOURCE: theCloud Storage URI.

For example, the following command loads thegs://mybucket/20180228T1256/default_namespace/kind_Book/default_namespace_kind_Book.export_metadataFirestore export file into a table named book_data.mybucket and mydataset were created in the US multi-region location.

bq --location=US load \--source_format=DATASTORE_BACKUP \mydataset.book_data \gs://mybucket/20180228T1256/default_namespace/kind_Book/default_namespace_kind_Book.export_metadata

API

Set the following properties to load Firestore export datausing the API.

1. Create a load job configuration that points to the source data inCloud Storage.

2. Specify your location in the locationproperty in the jobReference section of the job resource.

3. The sourceUris must be fully qualified, in the formatgs://BUCKET/OBJECT in theload job configuration. The file (object) namemust end in KIND_NAME.export_metadata. Only one URIis allowed for Firestore exports, and you cannot use a wildcard.

4. Specify the data format by setting the sourceFormat property toDATASTORE_BACKUP in the load job configuration. Datastore Backupis the correct option for Firestore. Firestore andDatastore share an export format.

5. To load only specific fields, set the projectionFields property.

   (Video) Import/Export Firebase Firestore Data of a project | Generate .json file of Firebase Firestore

6. If you are overwriting an existing table, specify the write dispositionby setting the writeDisposition property to WRITE_TRUNCATE.

Python

# TODO(developer): Set table_id to the ID of the table to create.table_id = "your-project.your_dataset.your_table_name"# TODO(developer): Set uri to the path of the kind export metadatauri = ( "gs://cloud-samples-data/bigquery/us-states" "/2021-07-02T16:04:48_70344/all_namespaces/kind_us-states" "/all_namespaces_kind_us-states.export_metadata")# TODO(developer): Set projection_fields to a list of document properties# to import. Leave unset or set to `None` for all fields.projection_fields = ["name", "post_abbr"]from google.cloud import bigquery# Construct a BigQuery client object.client = bigquery.Client()job_config = bigquery.LoadJobConfig( source_format=bigquery.SourceFormat.DATASTORE_BACKUP, projection_fields=projection_fields,)load_job = client.load_table_from_uri( uri, table_id, job_config=job_config) # Make an API request.load_job.result() # Waits for the job to complete.destination_table = client.get_table(table_id)print("Loaded {} rows.".format(destination_table.num_rows))