Skip to main content
All CollectionsData Management and Exporting
Large Upload Workaround
Large Upload Workaround

If you are having issues uploading extremely large datasets, here is a temporary workaround.

Josh Sinclair avatar
Written by Josh Sinclair
Updated over a year ago

NOTE: Using this method will incur costs from AWS for the temporary storage of your data. Make sure you delete the data once it has been retrieved by Trendspek to minimise costs. Trendspek is not responsible for these costs.

The Trendspek My Uploads page whilst robust, can still have issues with extremely large uploads. We are working to resolve this, but in the meantime you can use the below workaround. In a nutshell, we will be:

  1. Creating an AWS S3 Bucket

  2. Granting Trendspek access to this bucket by setting a bucket policy

  3. Uploading images to the S3 bucket

  4. Notifying Trendspek that a dataset is ready for processing

Begin by creating an AWS account if you do not already have one.

Creating an AWS S3 Bucket

  1. Sign in to the AWS Management Console and open the Amazon S3 console at https://console.aws.amazon.com/s3/.

  2. Choose Create bucket.

    The Create bucket wizard opens.

  3. In Bucket name, enter a DNS-compliant name for your bucket. We suggest using a dedicated bucket for interfacing with Trendspek, so for the name you might choose something like mycompany-trendspek-uploads

    Important

  4. In Region, choose the AWS Region where you want the bucket to reside.

    Choose a Region close to you to minimize latency.

  5. Choose Create bucket.

Granting Trendspek access to this bucket by setting a bucket policy

  1. Open the Amazon S3 console at https://console.aws.amazon.com/s3/.

  2. In the Buckets list, choose the name of the bucket that you created in the previous step.

  3. Choose Permissions.

  4. Under Bucket policy, choose Edit. This opens the Edit bucket policy page.

  5. In the Edit Statement box, paste the following. Make sure you change mycompany-trendspek-uploads to the bucket name you created above.

{
    "Version": "2012-10-17",
    "Id": "TrendspekUploadRetrieval",
    "Statement": [
        {
            "Sid": "Allow Trendspek to retrieve data from s3 bucket",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::573219254722:user/s3-sync"
            },
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::mycompany-trendspek-uploads/*",
                "arn:aws:s3:::mycompany-trendspek-uploads"
            ]
        }
    ]
}

Trendspek is now able to view and download anything stored in this bucket. Remove this bucket policy at any time if you wish to revoke our access.

Uploading images to the S3 bucket

Upload your imagery to s3://mycompany-trendspek-uploads/asset-name/

Technical users should opt to use the AWS S3 CLI sync function. It is extremely reliable in our experience.

Less technical users should use one of many available S3 clients to facilitate the upload. One of those is CrossFTP. Follow this guide to connect s3 to CrossFTP.
​
Other options include s3Browser.

Notifying Trendspek that a dataset is ready for processing

Send an email to [email protected] using the following template

New dataset ready from processing from AWS S3

Path to images: s3://mycompany-trendspek-uploads/asset-name/

Company Name:

Asset Name:

Capture Date:

Notes:

Did this answer your question?