An Ambitious Abandoned Project

Back in September, 2022, I was constantly downloading videos from Twitter using some random website. I noticed that it would be much better if only there was a "Download" button directly in each tweet, so I didn't have through the whole process of copying, pasting, and downloading the Tweet links in a third party website. I looked if there was any extensions that did what I was looking for, and I kind of found a couple, but they looked shady, and they haven't been updated in a long time.

And so my journey began

I learned to create a chrome extension in which I could execute some code. That step was not really hard, in fact, it reminded me of when I first created Cereza (my first Discord bot), and how I had to take the time to read documentation and follow tutorials. Overall, it was a great learning experience.
After creating the extension, I started investigating about the Twitter API.
The Twitter API has a variety of versions, but you can divide them in V1.1, and V2 (V2 being the latest one).

The V1.1 has many subdivisions, such as Essential, Elevated, and Academic Research.

Turns out that the only way to get a download link for a Twitter video, is using the API V1.1, and the only way to use the API V1.1, is applying to "V1.1 ACCESS" in your developer account (which I had to do), and when getting the tweet itself, you had to use the "extended_entities=true" parameter in the base URL. But of course you had to use all the tokens and keys that Twitter gives you when creating a developer account and when creating a Twitter app.

Before getting into the whole Twitter API thing, a took a whole Postman course in YouTube, which was actually critically useful, because as it turns out, Postman is a very useful tool when making API calls.
Here is the link to that tutorial in case you want to check it out (it's totally worth taking).

Having all my Twitter developer tokens and keys, I was able to perform some tests in Postman, and after some testing, I was able to understand what to look for, and how.

What I needed

Since the GET request that I'm going to use in the extension that I am creating (Which by the way, is called BirdClip) is not something small, I need some parameters to use when making the GET request.

These parameters are normally some tokens and keys to authenticate which developer account is doing these requests (in this case, mine, of course), and to authenticate which project of mine is doing these requests (in this case, I created an application in the Twitter developer portal named BirdClip; keep in mind that each Project can hold up to 3 applications in the free plan), and to authenticate which application of that Project is performing the request (in this case, I created an app called Rubí).

There are also temporal parameters, such as:
  • Timestamp, which is the amount of seconds that have elapsed since January 1st, 1970; for example the current timestamp as I'm writing this, would be 1679882775. The code used to get the timestamp would be:
    1
2
3
4
5
6
      function createTimestamp() {
    // Get the current time in seconds since the Unix epoch (1970-01-01)
    const time = Math.round(Date.now() / 1000);
    return time.toString();
  }
  var currenttimestamp = createTimestamp();

  • And nonce which is a random hexadecimal string, an example would be bveGl2yho4EV. The code to generate nonce would be:
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
      function generateNonce(length) {
      if (typeof length === "undefined") {
          length = 8;
      }
      if (length < 1) {
          console.warn("Invalid nonce length.");
      }
      var nonce = "";
      for (var i = 0; i < length; i++) {
          var character = Math.floor(Math.random() * 61);
          nonce += "0123456789ABCDEFGHIJKLMNOPQRSTUVWXTZabcdefghiklmnopqrstuvwxyz".substring(character, character + 1);
      };
      return nonce;
  }
  var readynonce = generateNonce(12);
The parameters mentioned above are pretty easy to get, but there are some others parameters that are way more complex to calculate.

The Signature

At the end of every OAuth 1.0 request (which is the authentication protocol that the Twitter API uses), there has to be a parameter that is called a signature, this signature is not a piece of cake to get.

First, you have to create a parameter string, which basically is all of the parameters that you are going to use in a single string, like the consumer key, the tweet id, the timestamp, the nonce, the access token, etcetera.

Then, you have to create a signature base string, which consists of the HTTP method you are using (in this case, GET), the percent encoded base URL, and the percent encoded parameter string.

In case you don't know what percent encoding is, it is basically replacing all symbols with a % and a code. For example, the percent encoding for the = symbol, would be %3D; the percent encoding for the & symbol, would be %26; the percent encoding for the ? symbol, would be $3F; and so on.
You can see all the symbol tables here.

After you create the signature base string, you have to create a signing key, which is the percent encoded consumer secret (one of the keys that the Twitter Developer Portal gives you), followed by an & symbol, and the percent encoded token secret (another of those keys).

And finally, to calculate the signature, you need to get the HMAC of the signature base string using the signing key, and using the SHA-1 hashing algorithm.
The output of the HMAC signing function is a binary string, so you need to encode it in base64 to get the signature that you will use at the end of the GET request.

Currently stuck

As of right now, I am stuck trying to generate the signature.
I am using CryptoJS to use the HmacSHA1 function, that takes the base string and the signing key as arguments to generate the signature, and it does generate a signature, but when trying to use said signature, it returns an error, citing some CORS policy.
After some tests, I discovered that this error also appears when some tokens and keys are wrong.
When making HTTP request, Postman also gives you the code that you can use to do that very same request in different programming languages, and when comparing Postman's code's parameters and the parameters that I have, the signature is the only one that doesn't match. But this doesn't necessarily mean that the signature is the only problem, because when using the pure code that Postman uses, I also get the same CORS policy error.
The thing is, that whether the signature is the only problem or not, it certainly doesn't match the signatures that Postman uses when using the exact same parameters (including timestamp and nonce).

Possible Explanations to the Problem

After several tests of different kinds, I can conclude that at least one of these options is the reason the signature doesn't match:
  • I am generating the base string wrongly.
  • I am generating the signing key wrongly.
  • CryptoJS's HMACSHA1 function is broken.
I've kept trying other methods, and still I can't come to a conclusion.
It has come to my attention that after testing two different HMAC generator websites, they return the same signature, which is different to the signature that the CryptoJS function returns.
This may possibly mean that the problem is indeed with CryptoJS, tomorrow I'll see if I can try another cryptography library, but for now, I'll go to sleep since it's really late.

The Next Day

So, I've tried encrypting a string with a key in CryptoJS and in online tools, and the result is that the online tools return the same output, but CryptoJS doesn't, I'm starting to think that the problem is either CryptoJS, or the way I'm using CryptoJS.
Ok, I've discovered something; when you use the HMAC SHA-1 function in CryptoJS, it returns a binary string (as expected), but the thing is what you convert that binary string to.

The website tools that I'm using, return a base16 string, and when I encode the CryptoJS binary string with the .enc.Base64 function, it returns a base64 string; to get the base16 string in CryptoJS (so it matches the websites' output), I just have to use the .toString() function.
There is also a website that provides a Base64 encoding output (which is the one that I have to use to get the final signature), and that output matches the one that the CryptoJS.enc.Base64 function returns, so I know that CryptoJS isn't broken.

But still, I don't know how to recreate the signature that Postman returns.

I've tried creating the signature with the parameters and keys that the Twitter documentation provided as an example, and it matches.

The mystery remains, I don't know where postman gets his signature from.

I'm reading a hell of OAuth 1.0 documentation, and for some reason, they are always plain blank text, and ugly.

I am thinking about giving up

I've read several documentations about OAuth1.0, and it seems like I do everything right, yet I can't match Postman's signature.

But apparently that's not the only and biggest problem.

When accessing external resources, you need to pass up something called CORS (Cross Origin Resource Sharing), where basically the domain that you are trying to get data from (in this case, api.twitter.com) has a whitelist of all the domains that are allowed to make requests to the Twitter API.
So, if you make a request, you need to specify from what domain it's coming from, (because you would be performing the request from the server side); but, if you are making it from a browser or a chrome extension (client side), you have to either specify "null" (for when it's from the browser) or the extension id (for when you're doing it from an extension).
The thing is, the Twitter API apparently doesn't allow server side requests (browsers or extensions) to make requests, because of how risky it is to deal with keys and tokens in the client side.

The Final Decision

It's been too many days, I've tried many things, and learned even more.
Sadly, it seems like the plans I had for BirdClip are not possible to achieve.
I really tried my best and looked for different ways to accomplish what I had in mind, to no result.
Even considering the amount of time put into this project, continuing with BirdClip is going to be inviable.
So, after careful consideration, I have no other option than to cancel this project completely.

Comments

Post a Comment

Popular posts from this blog

An Audioguide about the Geology of Ecuador

Image
Hello again. I've stayed a little AFK lately due to a big variety of reasons, such as my admission to college, a couple of life events, and some time away from home. But now I'm back. Today I'll write about a project I worked on during the first week of august. As you may have realized throughout this entire blog, all projects I write about are 100% mine; however, this project was not for me, but for my sister. She studies geology and had an assignment with creative freedom, so,  to finish her project on time,  I put Rubikraze temporarily aside (my current personal project). Like I said, my sister could choose any format for her assignment; the topic was geological formations of Ecuador. So, she asked me if I could make an HTML file with a map of Ecuador, and a list of audios (recorded by her of course) talking about different geological formations. As the good brother I am I said yes; the file in itself was not that complex or sophisticated, it was just an image of a map...
Read more

Active Projects as of Today

Hello again, I've been traveling quite a lot recently so I haven't had the opportunity to dedicate the necessary time and effort for writing  here and developing Rubikraze (the project I'm currently working on). This will be a small journal post just to talk about next goals and objectives in my projects and my occupations in general. Regarding Rubikraze, I'm still trying to find out whether the Color-Driven Summation (CDS) method I developed is useful to determine if a Rubik's Cube is solvable or not. My plan to use brute force by calculating the CDS of a bunch of cubes (both solvable and unsolvable) that I'll generate automatically using simple algorithms, and compare both kinds of CDS; therefore, if a solvable cube's CDS is the same as that of an unsolvable one, it would be evidence that my method doesn't work, so I would have to either completely scrap it or modify it. I'm quite optimistic that there has to be a numerical (yet not unique) harmony...
Read more

Explaining Color-Driven Summation

Image
One of my goals with Rubikraze is implementing a function that can solve any Rubik's cube thrown at it using the beginner's level method. In my previous post about Rubikraze I talked about how there are Rubik's cube configurations that are not possible to solve. For example, if you take a completely solved Rubik's cube, and rotate just one piece of it by manually taking it out and putting it back again, you'll end up with something like this: Assuming that's the only altered piece in the whole cube, we can consider that cube as impossible to solve. There's not a single pattern of movements that could possibly undo those changes without taking that piece out and putting it back again in the right orientation. With the example above it's pretty obvious that said cube is impossible to solve, but what if you took an unsolvable Rubik's cube, and mixed it? It'll still be unsolvable, and you know that because you knew it was unsolvable from the start, b...
Read more