This article looks at the Node implementation to stream content directly to google cloud storage, and we’ll look at how to convert that to a Cloud run function in a later one. Note that although I’m streaming videos, this technique could work with any kind of file.
In this project, I need to get videos from Vimeo to Google Cloud Storage in order to run the Video Intelligence API on them. There are a number of ways to do that, but I’d like to simply stream them directly using a cloud function.
Credentials
First, we need a service account, a bucketname and a folder that will be the target for the video files loaded from Vimeo. I’ve covered getting service accounts elsewhere so let’s assume you already have a downloaded service account .json file. I keep all my configs in one place, organized like this, with different configurations for different scenarios. These are referenced in the code so here’s a redacted version
ns.storage = {
kp: {
serviceAccountFile: "./creds/fxxxxxxd.json",
bucketName: "fxxxxxxs",
folderName: "fxxxxxxxxg"
},
pv: {
serviceAccountFile: "./creds/fyyyyyyyd.json",
bucketName: "fyyyyyyyys",
folderName: "fyyyyyyyyyyyg"
}
};
ns.getGcpCreds = () => ({
credentials: require(ns.storage[global.fidRunMode].serviceAccountFile),
bucketName: ns.storage[global.fidRunMode].bucketName,
folderName: ns.storage[global.fidRunMode].folderName
});
Vimeo download files
I have a note of these in my database, but for simplicity, I’ll just name the video file I want to load in my app. You’ll want the downloadable (as opposed to playable) distribution file. If like me, you plan to run the video intelligence API, the higher the quality, the more accurate the labelling will be.
Dependencies
There are not many – you’ll need to install them with npm or yarn.
const mime = require('mime-types');
const Storage = require('@google-cloud/storage').Storage;
const request = require('request');
Local dependencies
In my case, I’ll need to set the mode I’m running in (would normally do this through environment variables, but for this demo, I’m setting a global) and pick up my configuration secrets.
const request = require('request');
global.fidRunMode = "kp";
Setting up streaming from vimeo
The idea here is that we do an HTTP request to the Vimeo download link and immediately pipe the response to another stream which is connected to a cloud storage file.
/**
* stream video file direct from vimeo storage to google cloud storage
* @param {string} url the file to stream
* @param {stream} stream the stream to pipe it to
* @param {cb} function the callback when done
*/
const downloadFile = (url, stream , cb) => {
// request the video files
const sreq = request.get(url);
// check it worked
sreq.on('response', (response) => {
if (response.statusCode !== 200) {
return cb('Response error ' + response.statusCode);
}
// and pipe the output to the blob stream for cloud storage
sreq.pipe(stream);
});
// close when done
sreq.on('finish', () => stream.close(cb));
// need to report any errors
sreq.on('error', (err) => {
return cb(err.message);
});
};
Setting up streaming to Cloud Storage
By assigning a blob to a cloud storage file and associating a writestream with that blob, the data is magically piped from the input HTTP stream.
/**
* create the cloud storage stream
* the credentials/bucket name and filename are in the secrets file
* @param {string} fn the filename on cloud storage
*/
const createStorageStream = async (fn) => {
const type = mime.lookup(fn);
console.log('creating ', fn, type);
// get storage params
const creds = secrets.getGcpCreds();
const { credentials, bucketName, folderName } = creds;
const { projectId } = credentials;
const storage = new Storage({
projectId,
credentials
});
const bucket = storage.bucket(bucketName);
// we'll actually be streaming to this blob
const blob = bucket.file(folderName + '/' + fn);
const stream = blob.createWriteStream({
resumable: true,
contentType: type
});
// this stream will be piped to
return stream;
};
Kicking it off
Just give it a file name on cloud storage, and the input URL and that’s all there is to it.
// do the thing
(async () => {
// a downloadable vimeo link
const fn = 'https://player.vimeo.com/external/28xxxxxxxxxxx&download=1';
// create the target stream
const storageStream = await createStorageStream('40.mp4');
// get the file
downloadFile(fn, storageStream, (r) => {
console.log('created ', fn);
});
})();