Questions tagged [google-cloud-storage]

0 votes
1 replies
Using lifecycle policies in Google Cloud Storage, would imply early deletion charges if transition to Nearline to Coldline before 30 days?
I have enabled versioning in my bucket, old versions would be deleted 14 days after being uploaded. With my lifecycle configuration, on day 15 da...
asked 1 month ago
0 votes
2 replies
Dataflow job stucks while reading 35,000th file from google cloud storage
class Mp3_to_npyFn(beam.DoFn): def process(self, element): filename, e = element # get mp3 from the storage bucket =...
4 votes
1 replies
Version control for machine learning data set with large amount of images?
We starting to use dvc with git to control versioning of machine learning projects. For dvc remote storage we use google cloud storage. Our da...
0 votes
1 replies
How can I save a string to a textfile in Cloud Storage, in an angular app?
Here is my problem: I'm writing an angular app, I use App Engine and Google Cloud Storage as the backend (JAVA 8). Now I have text, and I want t...
2 votes
1 replies
Illegal unquoted character for Logstash's Google Cloud Storage input plugin
I am trying to download stackdriver logs from my GCP bucket using logstash with this configuration: google_cloud_storage { interval => 6...
0 votes
0 replies
GAE: getting error “java.util.concurrent.CancellationException: Task was cancelled.” while query in Google Storage
While querying in Google Storage entity, getting following error java.util.concurrent.CancellationException: Task was canceled. The query i...
3 votes
1 replies
IntelliJ can't find google-cloud-storage classes
I'm trying to use Google cloud storage and I'm following this guide: https://cloud.google.com/storage/docs/reference/libraries#using_the_client_l...
0 votes
1 replies
GCP, only list buckets where user has permissions
I am trying to figure out a way to allow a GCP user to list buckets but only those where the user has permissions (through ACL). The reason is be...
0 votes
2 replies
Upload Pandas dataframe as a JSON object in Cloud Storage
I have been trying to upload a Pandas dataframe to a JSON object in Cloud Storage using Cloud Function. Follwing is my code - def upload_blob(...
2 votes
1 replies
How to save uploaded file url in firebase using react
I am new to React Redux firebase. I want to upload the file and save the url in database. I'm able to save the file to storage, but cannot save t...
0 votes
2 replies
Airflow : pass parameter from python function to MySQL Operator
What I want to do : - compare missing ID from MySQL and BigQuery - get all data from MySQL where id in missing ID table = 'orders' def get_miss...
1 votes
0 replies
Get URL from firebase storage file
I want to get a downloadURL from a file inside my storage from within the onFinalize trigger. In a best case scenario, I want an URL as short as...
0 votes
0 replies
Is there any way to resolve this image loading exception caught in firebase storage?
I'm uploading a profile picture of the user which is stored in the firebase storage. Then in my app I'm retrieving the image from the firebase st...
0 votes
1 replies
How to optimise calling “sub” Cloud Functions in parallel from an HTTP triggered function
I have thousands of log files in a cloud storage bucket that I need to process and aggregate using an HTTP triggered cloud function and am lookin...
0 votes
0 replies
firebase cloud function using getSignedUrl for thumbnail is expiring after about a week, no clear reason why?
I'm able to successfully generate a signedUrl for my thumbnails I am creating, but after about a week they no longer work... I am not getting an...
-1 votes
2 replies
Move an entire bucket from google cloud storage to BigQuery?
From Google Cloud Storage to BigQuery transfers documentation which I have been reading, I can see how I can load data files one-by-one. Is the...
0 votes
1 replies
How to remove .jpg from file name with gsutil over 20000 files already in bucket
I have about 30000 images in bucket which many of have .jpg extension. How can I remove .jpg with gsutil from filenames? 0 Please see stac...
0 votes
1 replies
Google Cloud Storage `predefinedAcl` and `file.makePublic()` not working
I'm trying to get permanent download URLs when I upload files to Firebase Storage (Google Cloud Storage) from Firebase Cloud Functions (Google Cl...
0 votes
1 replies
Google Cloud Function - Error: Cannot find module google-cloud/bigquery
I am working to create a cloud function that moves avro files from GCS to BigQuery, anytime a new file lands in GCS. I am using the cloud functio...
1 votes
1 replies
Google Cloud Storage Multi-Regional bucket slow deletion
I am experiencing slow deletion for a GCS Multi-Regional bucket. I was wondering if this is normal performance to be expected since the bucket is...
-1 votes
1 replies
regional vs multi-regional buckets
In the project I see multi regional buckets created by someone by mistake and it is been used for the data pipelines. It should have been regiona...
asked 1 month ago
0 votes
1 replies
JSON API response incorrectly claims my API key has an IP restriction
I'm trying to upload files via curl with an API key. I created the API key with no application restrictions, but with API restrictions for storag...
0 votes
0 replies
How to fix stream reading past template in Google Cloud Storage?
I'm trying to upload a file (82 MB) to a regional gcloud bucket using python but I'm getting the following error: File "C:\PROGRA~2\Google\CLOUD...
asked 1 month ago
0 votes
1 replies
Is it possible to use my domain for the URL for publicly available objects in a Google Cloud Storage Bucket?
I set up a Google Cloud Storage bucket with a domain I own, after going through the ownership verification step. I was thinking I would be able...
asked 1 month ago
0 votes
1 replies
cloud composer-airflow throws error : Broken DAG: cannot import name '_parse_data' when importing new dag
I'm trying to created a DAG in Cloud Composer. When importing, I'm getting the following error: Broken DAG: [/home/airflow/gcs/dags/airflow_bigq...