The Nest
Explore
0
README.md

prerequisites

  • nodejs >= 14
  • imagemagick >= 7
  • jpegtran from libjpeg >= 9d, libjpeg-turbo >= 2.0.4 or mozjpeg >= 3.3.1
  • PostgreSQL >= 12
  • git >= 2.25
  • periods >= 1.1
  • pg_libphonenumber >= 753e2fa4be452620455a099aeda917648f2da70a
  • tor

first_last_agg installation if package not available

git clone https://github.com/wulczer/first_last_agg.git
cd first_last_agg
#apt-get install postgresql-server-dev-12 build-essential
make install #sudo if not root

range_agg installation if package not available

git clone https://github.com/pjungwir/range_agg.git
cd range_agg
#apt-get install postgresql-server-dev-12 build-essential
make install #sudo if not root

periods installation if package not available

git clone https://github.com/xocolatl/periods.git
cd periods
#apt-get install postgresql-server-dev-12 build-essential
make install #sudo if not root

pg_libphonenumber installation if package not available

git clone https://github.com/blm768/pg-libphonenumber
cd pg-libphonenumber
#apt-get install postgresql-server-dev-12 build-essential libphonenumber-dev
make install #sudo if not root

general setup

Locate the torrc file and append these two lines to it

NewCircuitPeriod 5
MaxCircuitDirtiness 30

This is to get a new IP address every 30 seconds so that the scrapers don't get blocked by Shopify(429)

alias node="node --unhandled-rejections=strict"
npm install
npm install nodemon -g #this is so the server will reload on change
psql -U postgres -f initialize.sql #sudo -u postgres psql -f initialize.sql if permission issue
cp dotenv .env
nano .env #if port 80 doesn't work, try 9001. get google maps and cloudinary credentials from kevin
nodemon server.js # or node server.js    can pass --inspect to nodemon or node and use chrome debugger :)

documentation

  • https://unetworking.github.io/uWebSockets.js/generated/
  • https://github.com/uNetworking/uWebSockets.js/blob/master/examples/WebSockets.js
  • https://github.com/uNetworking/uWebSockets.js/blob/master/examples/PubSub.js

request.json, response.json, event.json are the abstract contracts for the websocket protocol

update user_account set type = 'god' where id = 1; to promote an existing account.

to run a scrape of ocs.ca

cd scrape
node ocs.ca.js

in server.js, uWS.App().ws('/*', { message: function...}) parses the incoming message into JSON, following the abstract contract request.json and then is essentially a huge switch statement on what. to be refactored. It will respond according to the abstract contract response.json. For each branch of the switch statement, it usually follows the pattern of "is user authenticated?", "is user authorized?", "read" or "write", "return what was read" or "return success" or return an error".

database design

schema the history tables do not have any foreign keys because periods guarantees the validity.

updating a database

https://github.com/xocolatl/periods#altering-a-table-with-system-versioning

#for dev databases
psql -U postgres -c "create database clone with template lobomj"

#todo: https://serverfault.com/q/627169
#OR from production to local https://dba.stackexchange.com/q/82161
dropdb -U postgres clone
createdb -U postgres clone
pg_dump "port=<port> host=<host> user=<user> dbname=<db> sslcert=<cert> sslkey=<key> sslrootcert=<ca.crt> sslmode=verify-full" -d lobomj -Fc | pg_restore -U postgres -d clone

psql -U postgres -f initialize.sql -v name=diffme

pg_dump -U postgres clone --schema-only -f clone.sql
pg_dump -U postgres diffme --schema-only -f diffme.sql

#diff with your favourite tool
#write queries to alter database to update.sql
psql -U postgres -d clone -f update.sql

#run tests

#update dev
psql -U postgres -d lobomj -f update.sql

#OR propagate local changes to production
psql "port=<port> host=<host> user=<user> dbname=<db> sslcert=<cert> sslkey=<key> sslrootcert=<ca.crt> sslmode=verify-full" -d lobomj -f update.sql

dropdb -U postgres clone
dropdb -U postgres diffme

database backups

The lobomj production database is backed up at 6am GMT daily to bud.lobogene.com using a cron job (user ajay).

The last 30 backups are stored, and then rotated. Email notifications are sent to core dev team and info@lobogene.com.

backup config

  • (using backup gem): https://github.com/lobo-genetics/bud_lobogene/blob/master/lib/backup/models/lobomj_db_backup.rb
  • (using whenever gem for scheduling): https://github.com/lobo-genetics/bud_lobogene/blob/master/lib/backup/config/schedule.rb
  • (honeybadger checkin URL): https://app.honeybadger.io/projects/73866/check_ins/EWIROZ