Go Watch

A change detection server that can notify through various services, written in Go

Intro

GoWatch works through filters, a filter performs operations on the input it recieves.
Here is an example of a 'Watch' that calculates the lowest and average price of 4090s on NewEgg and notifies the user if the lowest price changed:

Note that everything, including scheduling/storing/notifying, is a filter.

Schedule is a cron filter with a '@every 15m' value, so this will run every 15 minutes.

NewEgg Fetch is a Get URL filter with a 'https://www.newegg.com/p/pl?N=100007709&d=4090&isdeptsrh=1&PageSize=96' value, it's output will be the HTTP response.

Select Price is a CSS filter with the value '.item-container .item-action strong[class!="item-buying-choices-price"]' value, it's output will be the html elements containing the prices.
An XPath filter could also have been used.

Sanitize is a Replace filter, using a regular expression ('[^0-9]') it removes anything that's not a number.

Avg is an Average filter, it calculates the average value of its inputs.

Min is a Minimum filter, it calculates the minimum value of its inputs.

Store Avg and Store Min are Store filters, they store its input values in the database.

Diff is a Different Than Last filter, only passing on the inputs that are different then the last value stored in the database.

Notify is a Notify filter, if there are any inputs to this filter, it will execute a template and send the result to a user defined 'notifier' (Telegram/Discord/Matrix/etc).

Run

Docker

Easiest way to get started is with the prebuilt docker image:

docker run \
    -p 8080:8080 \
    -v /host/path/to/config:/config \
    ghcr.io/broodjeaap/go-watch:master

This will start GoWatch with a SQLite database, stored in the /config directory, which is probably fine for most use cases.

To use another database, the environment variable GOWATCH_DATABASE_DSN can be used, for example with a PostgreSQL database:

version: "3"

services:
  app:
    build:
      context: .
      dockerfile: Dockerfile
      target: base
    container_name: go-watch
    environment:
    - GOWATCH_DATABASE_DSN=postgres://gorm:gorm@db:5432/gorm
    volumes:
    - /host/path/to/config:/config
    ports:
    - "8080:8080"
    depends_on:
      db:
        condition: service_healthy
  db:
    image: postgres:15
    environment:
    - POSTGRES_USER=gorm
    - POSTGRES_PASSWORD=gorm
    - POSTGRES_DB=gorm
    volumes:
    - /host/path/to/db:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}"]
      interval: 10s
      timeout: 5s
      retries: 5

Authentication

Go-Watch doesn't have built in authentication, but we can use a reverse proxy for that, for example through Traefik:

version: "3"

services:
  app:
    build:
      context: .
      dockerfile: Dockerfile
      target: base
    container_name: go-watch
    environment:
    - GOWATCH_DATABASE_DSN=postgres://gorm:gorm@db:5432/gorm
    volumes:
    - /host/path/to/config:/config
    ports:
    - "8181:8080"
    depends_on:
      db:
        condition: service_healthy
    labels:
    - "traefik.http.routers.gowatch.rule=Host(`192.168.178.254`)"
    - "traefik.http.routers.gowatch.middlewares=test-auth"
  db:
    image: postgres:15
    environment:
    - POSTGRES_USER=gorm
    - POSTGRES_PASSWORD=gorm
    - POSTGRES_DB=gorm
    volumes:
    - /host/path/to/db:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}"]
      interval: 10s
      timeout: 5s
      retries: 5
    depends_on:
    - proxy
  proxy:
    image: traefik:v2.9.6
    command: --providers.docker
    labels:
    - "traefik.http.middlewares.test-auth.basicauth.users=broodjeaap:$$2y$$10$$aUvoh7HNdt5tvf8PYMKaaOyCLD3Uel03JtEIPxFEBklJE62VX4rD6"
    ports:
    - "8080:80"
    volumes:
    - /var/run/docker.sock:/var/run/docker.sock

Change the Host label to the correct ip/hostname and generate a user/password string with htpasswd for the basicauth.users label, note that the $ character is escaped with $$

Filters

GoWatch comes with many filters that should, hopefully, be enough to allow for most use cases.

Cron

The cron filter is used to schedule when your watch will run.
It uses the cron package to schedule go routines, some common examples would be:

• @every 15m: will trigger every 15 minutes starting on server start.
• @hourly: will trigger every beginning of hour.
• 30 * * * *: will trigger every hour on the half hour.

For more detailed instructions you can check its documentation.

Get URL

Fetches the given URL and outputs the HTTP response.
For more complicated requests, POSTing/headers/login, use the HTTP functionality in the Lua filter.
During editing, http requests are cached, so not to trigger any DOS protection on your sources.

Get URLs

Fetches every URL given as input and outputs every HTTP response.
During editing, http requests are cached, so not to trigger any DOS protection on your sources.

CSS

Use a CSS selector to filter your http responses.
The Cascadia package is used for this filter, check the docs to see what is and isn't supported.

XPath

Use an XPath to filter your http responses.
The XPath package is used for this filter, check the docs to see what is and isn't supported.

JSON

Use a this to filter your JSON responses, the gjson package is used for this filter.
Some common examples would be:

• product.price
• items.3
• products.#.price

Replace

Simple replace filter, supports regular expressions.
If the With value is empty, it will just remove matching text.

Match

Matches the inputs to a value, supports regular expressions.

Substring

Substring allows for a Python like substring selection.
For the input string 'Hello World!':

• :5: Hello
• 6:: World!
• 6,0,7: WHo
• -6:: World!
• -6:,:5: World!Hello

Contains

Inputs pass if they contain the given value, basically same as Match but more basic.

Store

Stores each input value in the database under its own name, should probably limit this to single inputs (after Minimum/Maximum/Average filters).

Notify

Executes the given template and sends the resulting string as a message to the given notifier(s).
It uses the Golang templating language, the outputs of all the filters can be used by the name of the filters.
So if you have a Min filter like in the example, it can be referenced in the template by using {{ .Min }}.
The name of the watch is also included under .WatchName.

Math

Sum

Sums the inputs together, nonnumerical values are skipped.

Minimum

Outputs the lowest value of the inputs, nonnumerical values are skipped.

Maximum

Outputs the highest value of the inputs, nonnumerical values are skipped.

Average

Outputs the average of the inputs, nonnumerical values are skipped.

Count

Outputs the number of inputs.

Round

Outputs the inputs rounded to the given decimals, nonnumerical valuesa are skipped.

Condition

Different Than Last

Passes an input if it is different than the last stored value.

Lower Than Last

Passes an input if it is lower than the last stored value.

Lowest

Passes an input if it is lower than all previous stored values.

Lower Than

Passes an input if it is lower than a given value.

Higher Than Last

Passes an input if it is higher than the last stored value.

Highest

Passes an input if it is higher than all previous stored values.

Higher Than

Passes an input if it is higher than a given value.

Lua

The Lua filter wraps gopher-lua, with gopher-lua-libs to greatly extend the capabilities of the Lua VM.
A basic script that just passes all inputs to the output looks like this:

for i,input in pairs(inputs) do
	table.insert(outputs, input)
end

Both inputs and outputs are convenience tables provided by GoWatch to make Lua scripting a bit easier. There is also a logs table that can be used the same way as the outputs table (table.insert(logs, 'this will be logged')) to provide some basic logging.

Much of the functionality that is provided through individual filters in GoWatch can also be done from Lua.
The gopher-lua-libs provide an http lib, whose output can be parsed with the xmlpath or json libs and then filtered with a regular expression or some regular Lua scripting to then finally be turned into a ready to send notification through a template.

Dev

type script compilation

tsc static/*.ts --lib es2020,dom --watch --downlevelIteration

Dependencies

The following libaries are used in Go-Watch:

• Gin for HTTP server
  • multitemplate for template inheritance
• Cascadia for CSS selectors
• htmlquery for XPath selectors
• validator for user user input validation
• tgbotapi for Telegram
• cron for job scheduling
• viper for config management
• gjson for JSON selectors
• gopher-lua for Lua scripting
  • gopher-lua-libs for expanding the Lua scripting functionality
• net for http fetching
• gorm for database abstraction
  • sqlite
  • postgres
  • mysql
  • sqlserver
• bootstrap