From 45e747336ac5578e2ffd97cc00f41f3fd43f5093 Mon Sep 17 00:00:00 2001 From: Vaansh Lakhwara Date: Tue, 18 Jul 2023 01:28:18 -0400 Subject: [PATCH 1/4] docs: add license and documentation --- LICENSE | 21 +++++ README.md | 268 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 289 insertions(+) create mode 100644 LICENSE create mode 100644 README.md diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..21ce6b6 --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2023 Vaansh Lakhwara + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md new file mode 100644 index 0000000..1fd04b5 --- /dev/null +++ b/README.md @@ -0,0 +1,268 @@ + + + +

+ + + + + + + + + +

+ + +
+

+ + Logo + + +

GoRe

+ +

+ A Content Resharing Engin Written in Go. +
+ Explore the docs » +
+
+

+

+ + +
+
    +
  1. + About The Project + +
    2. +
  2. + System Design + +
    3. +
  3. + Overall System Architecture + +
    4. +
  4. Deployment
    5. + +
  5. Miscellaneous
    6. + + +
  6. License
    7. +
+
+ +## About The Project + +A few years ago I wrote a few scripts to automate posting content from Reddit to Instagram. Since then I've had the idea to capitalize on the rise of short form content – this is an implementation of that idea. I wanted a way to automate posting content from multiple sources – regardless of the platform (YouTube, Instagram, Tiktok, etc.) to another platform of my choice. Something scalable and well thought out based on my previous experience with this project. It's a project I have been meaning to work on from quite some time now and I'm happy with the way it has turned out so far. That being said it is still a prototype and you can track my progress [here](https://github.com/users/Vaansh/projects/1). + + + + + +### Built With + +

+ + + + + + +

+ + + +## System Design + +A `Task` makes up the basic structure of my application. It is defined as follows and consists of many publishers and one subscriber. Channels in Go seemed like the right message passing system to start with (it gave me the pub-sub mechanism I was looking for to implement something like this). Every task when created, has its own channel where the publishers write and subsciber consumes – making each component responsible for their own fetching or posting mechanism – allowing for separation of concerns. + +```go +type Task struct { + Id string + Publishers []publisher.Publisher + Subscriber subscriber.Subscriber + Quit chan struct{} // covered later +} +``` + +It has a `Run()` method which starts a goroutine for every publisher (that publish posts to the channel) and the subscirber (which subscribes to this channel and is responsible for ensuring: (1) ensuring uniqueness of the post (2), storing the file locally, uploading it to a cloud storage, (3) deleting them from both after posting it to the desired platform, and (4) ensuring it maintains a certain posting frequency). The definition of Publisher and Subscriber are outlined below for reference. + +```go +type Publisher interface { + PublishTo(c chan<- model.Post, quit <-chan struct{}) + GetPublisherId() string +} + +type Subscriber interface { + SubscribeTo(c <-chan model.Post) + GetSubscriberId() string +} +``` + +_NOTE:_ As of now, YouTube is the only kind of publisher and Instagram is the only kind of subscriber that is supported. Please follow the rest of the document with that in mind. + +There is also a `Quit` channel that simply exists for force quitting a task, this is for use by the `TaskService`. It is the service responsible for managing the lifecycle of each task by maintaing a map for the tasks currently running and the quit channel to invoke, if the task should be stopped. + + + +## Overall System Architecture + +With the main application logic out of the way, I'll cover the entire software architecture from a higher level. Below is the overall workflow of the project in its current state. Each component if briefly talked about below. + +### Persistence with PostgreSQL + +I needed some kind of persistence mechanism to ensure I do not post the same content twice (subscriber side logic) so I decided to go with Google Cloud Platform and created an instance there. I didn't use GORM or any other Object Relational Mapper since it didn't fit my needs and I was focussed on delivering the project ...before my cloud services credits expire. + +### Cloud Bucket Storage + +In order to use Meta's developer API for content publishing (subscriber side logic), I needed the videos to be stored locally (in the `data/` directory) and then hosted on a web server before hitting their Graph API endpoint to publish the video. For this reason, I chose cloud bucket storage as it gave me an easy way to manage storage with Go's client library and since I was already using their database services. + +### Cloud and Local Logging + +For app-wide logging, I chose to go with GCP again, since it would give me a centralized way of going over my logs. For development purposes, I usually log them locally (logs get saved into the `log/` directory). Moreover, I had plans of containerizing the application, so I needed a good way to monitor or debug it. Local and cloud logging options can be set at an application level through the use of environment variables. In production, I have cloud enabled and local logging disabled. + +### Web API with Gin + +Since I wanted to take as much of a hands-off approach to the project, I decided to build an API around it that would allow me to manage my tasks by interfacing with the TaskService. I used Gin once and was quite happy with the experience so I wrote my handlers and decided to go with it again. Admittedly though, the middleware in its current state is weak (just a token set as an environment variable on the web server that needs to be added in my web requests). + + + +## Deployment + +The only way to make changes to the `main` branch is by opening PRs. Once a PR is merged, the docker image builds, and if succesful, is moved to the DigitalOcean container registry which serves as the main hosting place for my artifacts. I also publish this as a private image on DockerHub, for my own future reference. The flow described is displayed in the image below. + +### GitHub Actions + +I have two GitHub actions in place: + +#### PR Linting + +One for – linting PRs and making sure no weird looking code gets committed to `main`. + +#### Docker Image + +Another one for building and pushing the docker image to the registry, acting as my CI/CD pipeline. It also runs a script that SSH into my droplet, replaces the to the newest docker image in the registry and starts running it. + +### DigitalOcean Droplet + +I tried Google Cloud Run, Kubernetes, & Compute Engine but nothing really suited what I was looking for. I decided to go with another platform and created a Droplet (VM) on DigitalOcean. I really liked my experience with DigitalOcean so far, it gave me a streamlined developer solution I was looking for. + + + +## Miscellaneous + +There are certain things I wanted to discuss but didn't fit into any of these topics, so I'm briefly going over them above. + +### Future Plans + +The project page is always up to track the [status](https://github.com/users/Vaansh/projects/1). + +Overall, I need to add unit tests (and run it on PRs), and I think I can to a better job with logging. But there are still a lot of other platforms I need to implement – for both publishers and subscribers. + +My application is also quite stateful, which might be something I would want to look into if I want to redesign it before moving forward with the project. I still like it because it helped me learn bridge some knowledge gaps I had in Go. + +### Environment Variables + +All required environment variables so far can be seen in the `.env-sample` file. (PS: having the actual file is not necessary, I just use it so I have my secrets in one place but you can just set those environment variables directly). For obvious reasons, the actual variables file itself isn't tracked and I generate it throguh GitHub actions – this way I don't have to set them each time I change the machine I deploy them on. But it also means my docker image must remain priavte so I ensure no one has access to my containers. Also there is a sample service account key credentials JSON file that is needed for accessing various GCP services through client libraries. + +### Directory Structure + +This is my overall directory structure with a brief explanation of each. + +```sh +. +├── .github # everything github related +│   ├── workflows # all my workflows +│   │   ├── cicd.yml # building and deploying action file +│   │   ├── reviewdog.yml # pr linting action file +├── Dockerfile # used to create docker image +├── LICENSE # software license +├── README.md # documentation +├── cmd # all main programs live here +│   └── api # api directory containing main program +│   └── main.go # the main program +├── data # directory used to store .mp4 files locally +├── gcloud-service-key-sample.json # sample service key credentials +├── go.mod # all project dependenciess +├── go.sum # go module checksum file +├── internal # all application specific logic +│   ├── api # everything related to the api +│   │   ├── handler.go # api handlers +│   │   ├── middleware.go # authorization middleware +│   │   ├── request.go # request dtos +│   │   └── response.go # response dtos +│   ├── config # all my config files +│   │   ├── database.go # reads and creates database config +│   │   ├── logger.go # reads and creates logger config +│   │   └── storage.go # reads and creates logger config +│   ├── domain # the core of my application +│   │   ├── service.go # task service +│   │   └── task.go # definiton and task behaviour +│   ├── gcloud # gcp module +│   │   ├── database.go # functions for cloud database +│   │   ├── logger.go # functions for cloud logger +│   │   └── storage.go # functions for cloud bucket storage +│   ├── http # all my http clients +│   │   ├── client.go # general client definition +│   │   ├── instagram.go # instagram platform client +│   │   └── youtube.go # youtube platform client +│   ├── model # all models used in my application +│   │   ├── metadata.go # metadata used for things like scheduling +│   │   ├── post.go # model defining post +│   │   └── user.go # model defining user (subscriber) +│   ├── publisher # all publisher logic +│   │   ├── publisher.go # general publisher interface +│   │   └── youtube.go # youtube publisher implementation +│   ├── repository # for all database operations +│   │   └── user.go # user repository operations +│   ├── subscriber # general subscriber interface +│   │   ├── instagram.go # instagram subscriber implementation +│   │   └── subscriber.go # general subscriber interface +│   └── util # util module +│   └── util.go # all my helper functions +├── log # local log directory +│   ├── error.log # sample error log file +│   ├── info.log # sample info log file +│   └── warning.log # sample warning log file +├── scripts # a bunch of simple scripts I use often +│   ├── clean.sh # deletes all log and data files +│   └── run.sh # builds and runs cmd/api/main.go +├── supported.go # a centralized list of supported platforms +``` + + + +## License + +Distributed under the MIT License. See `LICENSE` for more information. From 428d312193ce6f8ee3132b37777c81b63cff8aed Mon Sep 17 00:00:00 2001 From: Vaansh Lakhwara Date: Tue, 18 Jul 2023 02:54:03 -0400 Subject: [PATCH 2/4] docs: change sample gcp key name --- gcloud-service-key-sample.json => gcloud-key.json | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename gcloud-service-key-sample.json => gcloud-key.json (100%) diff --git a/gcloud-service-key-sample.json b/gcloud-key.json similarity index 100% rename from gcloud-service-key-sample.json rename to gcloud-key.json From 7e5276a832f18a97bca519a5332b522589823c56 Mon Sep 17 00:00:00 2001 From: Vaansh Lakhwara Date: Tue, 18 Jul 2023 03:18:09 -0400 Subject: [PATCH 3/4] docs: fin documentation --- README.md | 296 +++++++++++++++++++++++++++++++----------------------- 1 file changed, 170 insertions(+), 126 deletions(-) diff --git a/README.md b/README.md index 1fd04b5..3a7df92 100644 --- a/README.md +++ b/README.md @@ -8,87 +8,87 @@

- - - - - - - - - + + + + + + + + +


- - Logo - - -

GoRe

- -

- A Content Resharing Engin Written in Go. -
- Explore the docs » -
-
-

+ + Logo + +

GoRe

+

+ A Content Resharing Engine Written in Go. +
+ Explore the docs » +
+
+

-
    -
  1. - About The Project - -
    2. -
  2. - System Design - -
    3. -
  3. - Overall System Architecture +
      +
    1. + About The Project + +
      2. +
    2. + System Design + +
      3. +
    3. + Overall System Architecture + +
      4. +
    4. Deployment
      5. - -
    5. Deployment
      6. - -
    6. Miscellaneous
      7. - - -
    7. License
      8. -
    +
    4. +
  4. License
    5. +
## About The Project A few years ago I wrote a few scripts to automate posting content from Reddit to Instagram. Since then I've had the idea to capitalize on the rise of short form content – this is an implementation of that idea. I wanted a way to automate posting content from multiple sources – regardless of the platform (YouTube, Instagram, Tiktok, etc.) to another platform of my choice. Something scalable and well thought out based on my previous experience with this project. It's a project I have been meaning to work on from quite some time now and I'm happy with the way it has turned out so far. That being said it is still a prototype and you can track my progress [here](https://github.com/users/Vaansh/projects/1). - - - +

+ + + +

### Built With @@ -101,8 +101,6 @@ A few years ago I wrote a few scripts to automate posting content from Reddit to

- - ## System Design A `Task` makes up the basic structure of my application. It is defined as follows and consists of many publishers and one subscriber. Channels in Go seemed like the right message passing system to start with (it gave me the pub-sub mechanism I was looking for to implement something like this). Every task when created, has its own channel where the publishers write and subsciber consumes – making each component responsible for their own fetching or posting mechanism – allowing for separation of concerns. @@ -134,50 +132,94 @@ _NOTE:_ As of now, YouTube is the only kind of publisher and Instagram is the on There is also a `Quit` channel that simply exists for force quitting a task, this is for use by the `TaskService`. It is the service responsible for managing the lifecycle of each task by maintaing a map for the tasks currently running and the quit channel to invoke, if the task should be stopped. - - ## Overall System Architecture With the main application logic out of the way, I'll cover the entire software architecture from a higher level. Below is the overall workflow of the project in its current state. Each component if briefly talked about below. +

+ +

+ ### Persistence with PostgreSQL I needed some kind of persistence mechanism to ensure I do not post the same content twice (subscriber side logic) so I decided to go with Google Cloud Platform and created an instance there. I didn't use GORM or any other Object Relational Mapper since it didn't fit my needs and I was focussed on delivering the project ...before my cloud services credits expire. +

+ +

+ ### Cloud Bucket Storage In order to use Meta's developer API for content publishing (subscriber side logic), I needed the videos to be stored locally (in the `data/` directory) and then hosted on a web server before hitting their Graph API endpoint to publish the video. For this reason, I chose cloud bucket storage as it gave me an easy way to manage storage with Go's client library and since I was already using their database services. +

+ +

+ ### Cloud and Local Logging For app-wide logging, I chose to go with GCP again, since it would give me a centralized way of going over my logs. For development purposes, I usually log them locally (logs get saved into the `log/` directory). Moreover, I had plans of containerizing the application, so I needed a good way to monitor or debug it. Local and cloud logging options can be set at an application level through the use of environment variables. In production, I have cloud enabled and local logging disabled. +

+ +

+ ### Web API with Gin Since I wanted to take as much of a hands-off approach to the project, I decided to build an API around it that would allow me to manage my tasks by interfacing with the TaskService. I used Gin once and was quite happy with the experience so I wrote my handlers and decided to go with it again. Admittedly though, the middleware in its current state is weak (just a token set as an environment variable on the web server that needs to be added in my web requests). - +A sample `POST` request to `tasks/:platform` + +

+ +

+ +A sample `DELETE` request to `tasks/:platform/:id` + +

+ +

## Deployment The only way to make changes to the `main` branch is by opening PRs. Once a PR is merged, the docker image builds, and if succesful, is moved to the DigitalOcean container registry which serves as the main hosting place for my artifacts. I also publish this as a private image on DockerHub, for my own future reference. The flow described is displayed in the image below. +

+ +

+ ### GitHub Actions I have two GitHub actions in place: #### PR Linting +

+ +

+ One for – linting PRs and making sure no weird looking code gets committed to `main`. #### Docker Image +

+ +

+ Another one for building and pushing the docker image to the registry, acting as my CI/CD pipeline. It also runs a script that SSH into my droplet, replaces the to the newest docker image in the registry and starts running it. +

+ +

+ ### DigitalOcean Droplet I tried Google Cloud Run, Kubernetes, & Compute Engine but nothing really suited what I was looking for. I decided to go with another platform and created a Droplet (VM) on DigitalOcean. I really liked my experience with DigitalOcean so far, it gave me a streamlined developer solution I was looking for. +

+ +

+ ## Miscellaneous @@ -188,6 +230,10 @@ There are certain things I wanted to discuss but didn't fit into any of these to The project page is always up to track the [status](https://github.com/users/Vaansh/projects/1). +

+ +

+ Overall, I need to add unit tests (and run it on PRs), and I think I can to a better job with logging. But there are still a lot of other platforms I need to implement – for both publishers and subscribers. My application is also quite stateful, which might be something I would want to look into if I want to redesign it before moving forward with the project. I still like it because it helped me learn bridge some knowledge gaps I had in Go. @@ -202,67 +248,65 @@ This is my overall directory structure with a brief explanation of each. ```sh . -├── .github # everything github related -│   ├── workflows # all my workflows -│   │   ├── cicd.yml # building and deploying action file -│   │   ├── reviewdog.yml # pr linting action file -├── Dockerfile # used to create docker image -├── LICENSE # software license -├── README.md # documentation -├── cmd # all main programs live here -│   └── api # api directory containing main program -│   └── main.go # the main program -├── data # directory used to store .mp4 files locally -├── gcloud-service-key-sample.json # sample service key credentials -├── go.mod # all project dependenciess -├── go.sum # go module checksum file -├── internal # all application specific logic -│   ├── api # everything related to the api -│   │   ├── handler.go # api handlers -│   │   ├── middleware.go # authorization middleware -│   │   ├── request.go # request dtos -│   │   └── response.go # response dtos -│   ├── config # all my config files -│   │   ├── database.go # reads and creates database config -│   │   ├── logger.go # reads and creates logger config -│   │   └── storage.go # reads and creates logger config -│   ├── domain # the core of my application -│   │   ├── service.go # task service -│   │   └── task.go # definiton and task behaviour -│   ├── gcloud # gcp module -│   │   ├── database.go # functions for cloud database -│   │   ├── logger.go # functions for cloud logger -│   │   └── storage.go # functions for cloud bucket storage -│   ├── http # all my http clients -│   │   ├── client.go # general client definition -│   │   ├── instagram.go # instagram platform client -│   │   └── youtube.go # youtube platform client -│   ├── model # all models used in my application -│   │   ├── metadata.go # metadata used for things like scheduling -│   │   ├── post.go # model defining post -│   │   └── user.go # model defining user (subscriber) -│   ├── publisher # all publisher logic -│   │   ├── publisher.go # general publisher interface -│   │   └── youtube.go # youtube publisher implementation -│   ├── repository # for all database operations -│   │   └── user.go # user repository operations -│   ├── subscriber # general subscriber interface -│   │   ├── instagram.go # instagram subscriber implementation -│   │   └── subscriber.go # general subscriber interface -│   └── util # util module -│   └── util.go # all my helper functions -├── log # local log directory -│   ├── error.log # sample error log file -│   ├── info.log # sample info log file -│   └── warning.log # sample warning log file -├── scripts # a bunch of simple scripts I use often -│   ├── clean.sh # deletes all log and data files -│   └── run.sh # builds and runs cmd/api/main.go -├── supported.go # a centralized list of supported platforms +├── .github # everything github related +│   ├── workflows # all my workflows +│   │   ├── cicd.yml # building and deploying action file +│   │   ├── reviewdog.yml # pr linting action file +├── Dockerfile # used to create docker image +├── LICENSE # software license +├── README.md # documentation +├── cmd # all main programs live here +│   └── api # api directory containing main program +│   └── main.go # the main program +├── data # directory used to store .mp4 files locally +├── gcloud-key.json # sample service key credentials +├── go.mod # all project dependenciess +├── go.sum # go module checksum file +├── internal # all application specific logic +│   ├── api # everything related to the api +│   │   ├── handler.go # api handlers +│   │   ├── middleware.go # authorization middleware +│   │   ├── request.go # request dtos +│   │   └── response.go # response dtos +│   ├── config # all my config files +│   │   ├── database.go # reads and creates database config +│   │   ├── logger.go # reads and creates logger config +│   │   └── storage.go # reads and creates logger config +│   ├── domain # the core of my application +│   │   ├── service.go # task service +│   │   └── task.go # definiton and task behaviour +│   ├── gcloud # gcp module +│   │   ├── database.go # functions for cloud database +│   │   ├── logger.go # functions for cloud logger +│   │   └── storage.go # functions for cloud bucket storage +│   ├── http # all my http clients +│   │   ├── client.go # general client definition +│   │   ├── instagram.go # instagram platform client +│   │   └── youtube.go # youtube platform client +│   ├── model # all models used in my application +│   │   ├── metadata.go # metadata used for things like scheduling +│   │   ├── post.go # model defining post +│   │   └── user.go # model defining user (subscriber) +│   ├── publisher # all publisher logic +│   │   ├── publisher.go # general publisher interface +│   │   └── youtube.go # youtube publisher implementation +│   ├── repository # for all database operations +│   │   └── user.go # user repository operations +│   ├── subscriber # general subscriber interface +│   │   ├── instagram.go # instagram subscriber implementation +│   │   └── subscriber.go # general subscriber interface +│   └── util # util module +│   └── util.go # all my helper functions +├── log # local log directory +│   ├── error.log # sample error log file +│   ├── info.log # sample info log file +│   └── warning.log # sample warning log file +├── scripts # a bunch of simple scripts I use often +│   ├── clean.sh # deletes all log and data files +│   └── run.sh # builds and runs cmd/api/main.go +├── supported.go # a centralized list of supported platforms ``` - - ## License Distributed under the MIT License. See `LICENSE` for more information. From 55e33b6bbbe8a1addb0a032f8fbf386cc99fe822 Mon Sep 17 00:00:00 2001 From: Vaansh Lakhwara Date: Tue, 18 Jul 2023 03:27:47 -0400 Subject: [PATCH 4/4] docs: minor improvements to documentation --- README.md | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 3a7df92..638ee9b 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@

- + @@ -46,10 +46,6 @@

  • System Design -
  • Overall System Architecture @@ -57,7 +53,7 @@
  • Persistence with PostgreSQL
  • Cloud Bucket Storage
  • Cloud and Local Logging
    • -
  • Web API with Gin
    • +
  • Web API with Gin
  • Deployment
    • @@ -185,7 +181,7 @@ A sample `DELETE` request to `tasks/:platform/:id` The only way to make changes to the `main` branch is by opening PRs. Once a PR is merged, the docker image builds, and if succesful, is moved to the DigitalOcean container registry which serves as the main hosting place for my artifacts. I also publish this as a private image on DockerHub, for my own future reference. The flow described is displayed in the image below.

    - +

    ### GitHub Actions @@ -228,7 +224,7 @@ There are certain things I wanted to discuss but didn't fit into any of these to ### Future Plans -The project page is always up to track the [status](https://github.com/users/Vaansh/projects/1). +The [project page](https://github.com/users/Vaansh/projects/1) is the best way to track future plans and current progress of the project.