Deliver remote media files | Cloudinary - The first knowledge sharing application in Vietnam

Auto-Upload and Fetch are two options for automatically fetching (retrieving) files from existing remote locations and delivering them using dynamic URLs. In both cases, the files are delivered through a powerful CDN network, greatly improving your users’ browsing experience, reducing load on your servers and lowering hosting costs, while you also benefit from on-the-fly image and video transformation and optimization, and all of this with minimum changes on your side.

The main difference between these two features is how the remote asset is handled:

Fetch enables on-the-fly transformation of remote images and videos with optimized delivery via a CDN. Fetched assets are cached in your Cloudinary product environment for performance reasons. Paid Cloudinary customers can request to have the remote assets checked on a regular basis (every 7 days by default) and if the remote asset has changed, the cached asset is updated accordingly. This feature supports image and video assets.
Auto-Upload also enables on-the-fly transformation of existing remote assets with optimized delivery via a CDN, while simultaneously uploading the fetched media asset to your Cloudinary product environment for further management, and thus benefiting from a variety of additional features (just like any other asset uploaded to your Cloudinary product environment). After an asset has been auto-uploaded from the remote location, all requests for the asset will reference the asset stored in your Cloudinary product environment. This feature supports image, video and raw assets.

The following table summarizes how these differences can impact different elements of the media management process:

Auto-Upload
Fetch

Management
Managed by Cloudinary just like any other uploaded asset (thus benefiting from additional features).
Managed on your side and just transformed and delivered by Cloudinary

Availability management
Cloudinary stores your auto-uploaded assets, including backups, historical revisions, etc.
Stored with Cloudinary for caching purposes only, which means that you need to continue to support your existing asset hosting solution. Caching counts against your storage quota.

Asset types
Supports any asset type; just make sure the resource_type value matches the type of asset you are retrieving.
Supports images and videos.

Eager transformations
You can use eager transformations to create multiple versions of the same asset during auto-upload rather than on the fly. This is especially valuable for transforming videos, large asset files, and complex transformations (e.g. transforming animated GIFs).
Not relevant

Upload presets
You can use upload presets to automatically apply actions to all assets uploaded to a specific auto-upload mapped folder by creating an upload preset with the same name as an auto-upload mapped folder. This enables you to eagerly create derived/transformed assets, request notifications, apply tagging, moderation and more.
Not relevant

SEO-friendly short URLs
Auto-uploaded URLs do not contain the lengthy URL of the original file.
Not relevant

Access permissions
You can define auto-uploaded assets as authenticated or private.
Not relevant

Note

Fetched URL box in the Restricted media types section of the Security page in your Console Settings. Otherwise, you need to

When fetching assets using your Cloudinary product environment, ensure that fetched URLs are unrestricted by clearing thebox in thesection of thepage in your Console Settings. Otherwise, you need to sign the URL

Allow fetched assets setting

The fetch feature is a quick way to deliver assets from remote URLs. The asset can be transformed and optimized on the fly, before being cached and delivered through fast, localized CDNs and not via local web servers. To create a fetch URL, simply prepend the following prefix to the URL of the asset:

(For customers with a Custom Domain Name (CNAME – available for Cloudinary’s Advanced plan and higher), prepend:
https://<your custom domain name>/image/fetch/ for images or https://<your custom domain name>/video/fetch/ for videos.)

For example: Assuming the following image of Benedict Cumberbatch from Wikipedia is also displayed on your website:

Access the following URL to fetch the image via Cloudinary:

Fetch my image

Both URLs return the exact same image, only the second one is cached and delivered through fast, localized CDNs and not via the local web server.

Tip

fetch delivery type to deliver profile pictures from various social media applications like Facebook, Twitter and others, you can accomplish this task more easily using specially defined social media delivery types along with the profile picture ID or name to accomplish this. For details, see

Although you can technically also use thedelivery type to deliver profile pictures from various social media applications like Facebook, Twitter and others, you can accomplish this task more easily using specially defined social media delivery types along with the profile picture ID or name to accomplish this. For details, see Social media profile pictures

You can also use the fetch feature to apply any of Cloudinary’s image and video transformations to the delivered asset. Simply add the transformation parameters to the URL directly after the fetch prefix and before the URL of the asset.

The following Cloudinary URL delivers an image of Benedict Cumberbatch from Wikipedia that has been transformed as follows: the width and height parameters are both set to 300 pixels, the image is cropped to fill in the new dimensions, retaining the original proportions and setting gravity to the detected face; the image is also made circular by setting the radius parameter to maximum, and finally, depending on the browser/device, the best image format is automatically delivered, optimized and cached via CDN:

Fetch image with transformations

The following Cloudinary URL delivers a video of the 2015 Tour De France from Wikimedia that has been transformed as follows: the width and height parameters are set to 600 and 400 pixels, and the video is resized to fit in the new dimensions, retaining the original proportions; a watermark is added by applying cloudinary_new_logo as an overlay, scaled down to 200 pixels and place in the north east corner of the video, 10 pixels away from the top and right; and finally, the video format is converted to the MP4 format:

Notes

If the asset referenced by the original URL changes, Cloudinary will not automatically update the fetched asset embedded in your site and all its transformed versions. For details, see Refreshing or deleting fetched assets.
Depending on your plan, there may be a limit to the size of the videos you can fetch. To find the relevant Usage Limit for your account, see the right panel of your Account Settings page.
Fetched video transformations aren’t supported by some of the SDKs with action-based syntax.

URLs containing special characters (particularly the ? character) need to be modified (or escaped) for use with the fetch feature. This is relevant for any special characters that are not allowed “as is” in a valid URL path, as well as other special unicode characters. These URLs should be escaped using % based URL encoding to ensure the URL is valid. Either escape the individual special characters themselves within the URL (for example, replacing ? with %3F) or escape the whole URL.

Note that the methods within Cloudinary’s client libraries (e.g. Ruby on Rails cl_image_tag method) automatically perform smart escaping, leaving the URL as simple as possible while escaping certain special characters. Manually escaping URLs as explained here is only relevant when simply adding the fetch prefix to existing URLs.

For example, the original URL:

Fully escaped becomes:

Escaped fetch URL

Fetch URL with minimal escaping:

You can restrict the list of allowed domains that can be used with the fetch feature in the Allowed fetch domains section on the Security page of the Console Settings.

Note

Fetched URL box in the Restricted media types section of the Security tab. This is also done automatically if fetch is not used within 7 days of account creation, in order to prevent abuse of the system for customers not using the functionality.

You can disable fetching by ticking thebox in thesection of thetab. This is also done automatically if fetch is not used within 7 days of account creation, in order to prevent abuse of the system for customers not using the functionality.

Restrict fetched images setting

When you fetch a remote asset, a copy of that asset as well as all transformations derived from the remote asset, are cached in your Cloudinary product environment for performance reasons. The following sections describe the available options for deleting and refreshing those assets.

Customers on a paid plan can request full cleanups of all fetched assets that occur at a set time interval. When a cleanup interval is defined, all of your fetched assets, as well as all transformations derived from them, are completely deleted from the cache every X days.

If one of these assets is requested again after the cleanup, it’s fetched again from the remote source. This feature is useful for cleaning up your storage in cases where many of your fetched assets become obsolete after a short time period, for example, banner ads.

You can request to set a custom cleanup interval by submitting a support request.

By default, Cloudinary checks one time after 7 days to see whether the remote images cached in your Cloudinary product environment have changed (by comparing the etag value). If it has changed, the cached image is replaced with the new one, and the next time that image, or a transformation of that asset, is requested, the updated version is delivered.

Cloudinary customers with a paid account can request to change this default from a single 7-day refresh to a different desired repeating interval.

You can request to set a custom fetch refresh interval by submitting a support request.

Note

When your refresh interval gets updated, that setting is applied only to images fetched after the setting change.

If you need to apply the new setting to fetched images that have already been delivered, you’ll need to delete the relevant fetched images from your product environment. Then, the next time any of those fetched images is requested, it will be re-fetched and the new refresh interval settings will be applied to it.

The auto-upload feature combines the advantages of dynamic fetching from existing online locations, with the advantages of managing media assets directly in your product environment (listing, editing, browsing, tagging, transforming, moderating, dynamically transforming, viewing reports, and more) via the Cloudinary Console. The auto-upload feature enables on-the-fly transformation of existing remote media assets and optimized delivery via a CDN, while automatically uploading the file to your Cloudinary product environment for further management and transformation.

The auto-upload feature is implemented by mapping a base remote URL to a specified folder in your Cloudinary Media Library. Then, whenever accessing a Cloudinary delivery URL containing the folder prefix, the media assets are automatically retrieved from the mapped URL if they are not already uploaded to the folder, and are available for further transformation, management and CDN delivery like any other asset uploaded to Cloudinary.

Mapping a remote URL prefix also allows your Cloudinary delivery URLs to be shorter and more SEO friendly, gives you fine-grained control on which remote domains are allowed for automatic uploads, enables lazy migration of all your existing assets to the cloud as well as other powerful origin pulling options. After an asset has been auto-uploaded from the remote location, all requests for the asset will reference the asset stored in your Cloudinary product environment.

The standard auto-upload URL takes the following structure:

For example: Creating a folder called remote_media and then mapping it to the URL prefix https://upload.wikimedia.org/wikipedia/ allows you to generate a Cloudinary delivery URL that substitutes the remote_media folder prefix for the URL prefix. When the Cloudinary delivery URL is accessed for the first time, Cloudinary automatically retrieves the remote asset from https://upload.wikimedia.org/wikipedia/ and stores it in your Cloudinary product environment.

To retrieve the following Olympic flag image from Wikimedia commons, and automatically upload it to your Cloudinary product environment:

Access the following Cloudinary URL:

Auto-upload remote image

The image is dynamically retrieved the first time this URL is accessed and stored in your Cloudinary product environment with a public ID of remote_media/commons/a/ae/Olympic_flag.

The auto-upload feature is configured in the auto-upload mapping section on the Upload page of the Console Settings:

Fill in the Folder name or path for the auto-upload mapping e.g. remote_media/remote_assets. The folder path can be up to 3 folders deep.
Fill in a URL prefix that will be mapped to the specified folder e.g. https://upload.wikimedia.org/wikipedia/ or s3://my-bucket/my-path/
To use a private storage (Amazon S3 or Google Storage) bucket as the remote URL prefix, you must make sure your bucket is whitelisted for your Cloudinary product environment. This is especially useful for lazy migration.
Click the Save button at the bottom of the page.

Once saved, any media stored under the directory you set as the URL prefix and requested by a Cloudinary URL that contains the remote_media folder in the Public ID, will automatically be retrieved and stored in your Cloudinary product environment. This asset is treated just like any other asset that was uploaded to your Cloudinary product environment.

Notes and guidelines for auto-uploading

You can map multiple folder names, each to a different remote URL prefix, by clicking the Add another mapping link in the Auto-upload mapping settings.
Instead of specifying a sub-folder, you can optionally map your root Cloudinary folder to a URL prefix by filling in a backslash for the folder name (/).

You can also use the auto-upload feature to apply any of Cloudinary’s image or video transformations on the fly to the delivered media asset. Simply add the transformation parameters to the URL directly before the mapped folder name.

For example, the following Cloudinary delivery URL retrieves an image from Wikimedia and stores it in Cloudinary’s demo product environment, then crops a thumbnail of the original image to a width of 200 pixels and a height of 200 pixels, centers the image on the detected face, makes it circular, applies a sharpen effect – and then delivers that modified image optimized and cached via CDN:

transformed auto-upload image

Auto-uploading from within your code can be easily accomplished using Cloudinary’s SDKs for all popular development frameworks. The principle is exactly the same as with any URL/tag building of an uploaded asset, except in this case the public ID does not have to match an existing asset, but can match a mapped folder name of a remote existing asset that will be implicitly fetched on first access. For example, the following code snippet retrieves the same transformed image as shown above:

Generally, when migrating to Cloudinary, customers upload all their media files via Cloudinary’s upload API and then update their web or mobile app to upload new files to Cloudinary.

The auto-upload feature offers a powerful alternative method to migrate media files within your own existing web application to your Cloudinary product environment, by dynamically uploading them only when first accessed rather than pre-uploading them. This “lazy migration” is especially useful if your site has lots of media assets, some of which may be very rarely (or never) accessed.

Auto-uploaded URLs share the exact same URL convention as Cloudinary’s regular uploaded asset URLs, which means that for each asset accessed, Cloudinary first checks whether a asset with the given public ID exists in your Cloudinary product environment, and if it doesn’t (and it matches a given auto-upload folder mapping), it is retrieved on the fly from the remote host.

The auto-upload examples given so far allow public access to the original asset as well as all its derived versions. As an alternative, you can control access to auto-uploaded remote assets by defining them as delivery type private or authenticated, and including the sign_url parameter set to true (required for authenticated assets).

For example, to auto-upload an image as authenticated, set the delivery type (type parameter) as authenticated and include a signature in the delivery URL. (In SDKs, set the sign_url parameter as true.) This setting prevents access to the image without an authentication signature:

Besides restricting an asset based on its delivery type, you can also globally control the behavior of the auto-upload feature so that auto-uploaded dynamic URLs require a signature in all cases, even for public assets of type upload:

In the Cloudinary Console, access the Security page of the Console Settings and then check the Uploaded option under the Restricted image types section.

For example, if the auto-upload feature is set as restricted, then the following code snippet allows auto-uploading by signing the URL (In SDKs, set the sign_url parameter to true):

When you use Cloudinary’s upload API you can leverage many powerful features (eagerly created derived/transformed assets, notifications on upload, adding tags, contextual and structured metadata, requesting moderation and more). To be able to benefit from the same features together with auto-upload, you can use upload presets for defining the automatic upload actions. Upload presets include one or more upload parameters, and any of Cloudinary’s upload parameters can be defined and included in a preset.

To instruct a particular upload mapping to use a particular upload preset, set the name of the upload preset to be the same name as the auto-upload mapped folder.

For example, naming an auto-upload mapped folder as remote_media and also naming an upload preset as remote_media will automatically apply all the actions specified in the upload preset when auto-uploading images to the remote_media folder.

Note

Upload preset names cannot contain slashes. Therefore, this option is supported only for auto-uploads mapped to top-level folders.

You can easily display thumbnail images of videos from various public video sites by referencing the unique video identifier used on their site. The cloudinary image delivery URL for the thumbnail follows the format:

https://res.cloudinary.com/<cloud name>/image/<type>/<identifier>.<format file extension>

Where:

cloud name – your product environment’s cloud name, a unique public identifier for URL building and API access.
type – the video website delivery type (youtube, hulu, vimeo, animoto, worldstarhiphop or dailymotion)
public ID – the unique identifier of the asset or the full URL of the video.
format file extension – the requested delivery format of the image.

For example, to display the thumbnail of the YouTube video: https://www.youtube.com/watch?v=o-urnlaJpOA, add the video ID to the URL (‘o-urnlaJpOA’ in this example):

youtube video thumbnail

The thumbnail can be resized and transformed like any other image uploaded to Cloudinary. For example, to scale the image to a width of 300 pixels with a 5px black border:

youtube video thumbnail scaled down with a border added

You can also pass the full URL of the video instead of just its identifier. The following example delivers a thumbnail of a YouTube video based on a full video URL (https://www.youtube.com/watch?v=aNwnPElsJGE). Note that if the URL includes special characters, they should be escaped (e.g., ‘%3F’ instead of ‘?’). If you use our client library helper methods then the escaping is done for you automatically.

youtube video thumbnail by full URL