NotAShelf/go-cdn
1
0
Fork 0
mirror of https://github.com/NotAShelf/go-cdn.git synced 2024-11-22 16:00:43 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Merge pull request #1 from NotAShelf/v2

Browse source 
rewrite the CDN to be more verbose and flexible
This commit is contained in:
NotAShelf 2023-06-03 22:24:21 +03:00 committed by GitHub
commit b249bfd141
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
5 changed files with 256 additions and 240 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

138
README.md
View file

@ -1,12 +1,23 @@
# go-cdn # go-cdn
An experimental CDN project in Go. > An experimental CDN project in Go.
> A lightweight server that allows you to upload and download files via HTTP. It provides optional authentication, input validation, error handling, and verbose logging. The server is implemented in Go and can be easily configured using a config.json file.
## Features ## Features
- Serve static files securely over HTTP - Serve static files securely over HTTP
- Basic authentication support - Basic authentication support
- File upload functionality - File upload & download functionality
### Error Handling
The server is equipped with error handling to handle various scenarios, including invalid requests, authentication failures, and payload size exceeding the maximum allowed limit. If an error occurs, the server will return an appropriate HTTP status code along with an error message.
Logging
The server uses the logrus library for logging. Verbose logging is enabled by default and displays detailed information about incoming requests, errors, and server restarts. The logs are printed to the console.
Customization
The server implementation provided is a starting point and may require customization based on your specific requirements. You can modify the handleGet and handlePost functions in the CDNHandler struct to implement your desired file upload and download logic.
## Possible use cases ## Possible use cases
@ -18,64 +29,107 @@ The CDN can be used in various scenarios, such as:
## Usage ## Usage
### Starting the CDN ### Prerequisites
1. Build the program - Go programming language (version 1.16 or later)
```go ### Installation
go build
```
2. Run the built binary 1. Clone the repository:
```go
```
### Using the CDN
To request a file from the CDN, use the following URL format:
```bash ```bash
http://your-cdn-server:port/file-path
git clone https://github.com/notashelf/go-cdn.git
``` ```
> Replace your-cdn-server with the hostname or IP address of your CDN server and port with the port number on which the server is running. Append the desired file path after the hostname and port. 2. Change to the project directory:
**Example:**
- `http://localhost:8080/images/file_you_have_uploaded.png`
- If the file exists and the request is authorized, the file will be served by the CDN.
- If the file doesn't exist, a 404 Not Found response will be returned.
#### Uploading Files to the CDN
> Send a POST request to the /upload endpoint of the CDN server.
Example using cURL:
```bash ```bash
curl -X POST -u username:password -F "file=@/path/to/file" http://your-cdn-server:port/upload cd your-repo
``` ```
> Replace username and password with the authentication credentials you have set in the main.go file. Replace your-cdn-server with the hostname or IP address of your CDN server and port with the port number on which the server is running. Provide the file path after the @ symbol in the -F parameter. 3. Install the dependencies:
**Example:**
```bash ```bash
curl -X POST -u admin:password -F "file=@/absolute/path/to/image.jpg" http://localhost:8080/upload go get github.com/sirupsen/logrus
go get github.com/pkg/errors
``` ```
- The uploaded file will be saved in the specified uploadPath directory. ### Configuration
> Note: The server responds with a success message if the file upload is successful. The server can be configured using the config.json file. Create a file named config.json in the same directory as the main file (main.go) and configure the following properties:
### Security Considerations - port (string): The port on which the server will listen for incoming connections.
- max_upload_size (integer): The maximum allowed size of file uploads, in bytes.
- heartbeat (string): The duration after which the server will automatically restart. Specify a value in the format "5m" for 5 minutes, "1h" for 1 hour, etc. Set to "0" to disable automatic restarts.
- require_auth (boolean): Whether to require authentication for file uploads and downloads.
- auth_username (string): The username for authentication (only applicable if require_auth is set to true).
- auth_password (string): The password for authentication (only applicable if require_auth is set to true).
It is highly recommended to use SSL/TLS encryption (HTTPS) for secure communication between clients and the CDN server. Example config.json file:
Change the default username and password in the main.go file to strong and secure credentials.
Consider implementing additional security measures based on your specific requirements. ```json
{
"port": "8080",
"max_upload_size": 10485760,
"heartbeat": "1h",
"require_auth": true,
"auth_username": "your-username",
"auth_password": "your-password"
}
```
### Usage
Start the CDN server by running the following command:
```bash
go run main.go -config config.json
```
The server will start and listen on the specified port. Verbose log messages will be displayed in the console.
#### Uploading a file:
Use the curl command to upload a file to the CDN server:
```bash
curl -X POST -F "file=@/path/to/file" http://localhost:8080
```
_Replace /path/to/file with the actual path to the file you want to upload. If authentication is enabled, provide the username and password when prompted._
#### Downloading a file:
Use the curl command to download a file from the CDN server:
```bash
curl -O http://localhost:8080/<filename>
```
_Replace <filename> with the name of the file you want to download._
### Authenticated Upload and Download
To perform an authenticated upload or download, you can use the following curl commands:
#### Uploading a File:
```bash
curl -X POST -F "file=@/path/to/file" -u "your-username:your-password" http://localhost:8080
```
_Replace /path/to/file with the actual path to the file you want to upload. The -u flag is used to provide the authentication credentials._
#### Downloading a File:
```bash
curl -O -u "your-username:your-password" http://localhost:8080/<filename>
```
Replace <filename> with the name of the file you want to download. The -u flag is used to provide the authentication credentials.
**Please note that these examples assume you're running the server on localhost with the specified port and authentication credentials. Make sure to adjust the hostname and port accordingly.**
# License # License

6
example_config.json
View file

@ -1,6 +0,0 @@
{
"username": "admin",
"password": "password",
"service_port": "8080",
"uploads_dir": "uploads/"
}

4
go.mod
View file

@ -3,7 +3,7 @@ module cdn
go 1.20 go 1.20
require ( require (
github.com/go-chi/chi v1.5.4 // indirect github.com/pkg/errors v0.9.1 // indirect
github.com/sirupsen/logrus v1.9.2 // indirect github.com/sirupsen/logrus v1.9.2 // indirect
golang.org/x/sys v0.8.0 // indirect golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8 // indirect
) )

7
go.sum
View file

@ -1,14 +1,13 @@
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/go-chi/chi v1.5.4 h1:QHdzF2szwjqVV4wmByUnTcsbIg7UGaQ0tPF2t5GcAIs= github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/go-chi/chi v1.5.4/go.mod h1:uaf8YgoFazUOkPBG7fxPftUylNumIev9awIWOENIuEg= github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/sirupsen/logrus v1.9.2 h1:oxx1eChJGI6Uks2ZC4W1zpLlVgqB8ner4EuQwV4Ik1Y= github.com/sirupsen/logrus v1.9.2 h1:oxx1eChJGI6Uks2ZC4W1zpLlVgqB8ner4EuQwV4Ik1Y=
github.com/sirupsen/logrus v1.9.2/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ= github.com/sirupsen/logrus v1.9.2/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8 h1:0A+M6Uqn+Eje4kHMK80dtF3JCXC4ykBgQG4Fe06QRhQ=
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.8.0 h1:EBmGv8NaZBZTWvrbjNoL6HVt+IVy3QDQpJs7VRIw3tU=
golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

341
main.go
View file

@ -2,236 +2,205 @@ package main
import ( import (
"encoding/json" "encoding/json"
"flag"
"fmt" "fmt"
"io" "io"
"log"
"net/http" "net/http"
"os" "os"
"path/filepath" "path/filepath"
"regexp" "time"
"runtime"
"strings"
"github.com/go-chi/chi"
"github.com/go-chi/chi/middleware"
"github.com/sirupsen/logrus" "github.com/sirupsen/logrus"
) )
// Config represents the configuration structure
type Config struct { type Config struct {
Username string `json:"username"` Port string `json:"port"`
Password string `json:"password"` MaxUploadSize int64 `json:"max_upload_size"`
ServicePort string `json:"service_port"` Heartbeat Duration `json:"heartbeat"`
UploadsDir string `json:"uploads_dir"` RequireAuth bool `json:"require_auth"`
AuthUsername string `json:"auth_username"`
AuthPassword string `json:"auth_password"`
UploadDirectory string `json:"upload_directory"`
} }
var ( // Duration is a custom type for decoding time.Duration from JSON
config Config type Duration time.Duration
logger *logrus.Logger
filenameRegex *regexp.Regexp
)
func main() { // UnmarshalJSON unmarshals a JSON string into a Duration
// Initialize logger func (d *Duration) UnmarshalJSON(data []byte) error {
logger = logrus.New() var durationStr string
logger.Formatter = &logrus.TextFormatter{ if err := json.Unmarshal(data, &durationStr); err != nil {
DisableTimestamp: false, return fmt.Errorf("error decoding Duration: %w", err)
FullTimestamp: true,
} }
// Load configuration parsedDuration, err := time.ParseDuration(durationStr)
err := loadConfig("config.json")
if err != nil { if err != nil {
logger.Fatalf("Error loading config file: %s", err) return fmt.Errorf("error parsing Duration: %w", err)
}
// Initialize filename regular expression
filenameRegex = regexp.MustCompile(`^[a-zA-Z0-9-_\.]+$`)
// Initialize router
router := chi.NewRouter()
router.Use(middleware.Logger)
router.HandleFunc("/", serveCDN)
router.HandleFunc("/upload", handleUpload)
// Start server
address := fmt.Sprintf(":%s", config.ServicePort)
logger.Infof("Starting CDN server on port %s...", config.ServicePort)
logger.Infof("Serving files from %s", config.UploadsDir)
// Print upload path and list files
uploadPath := filepath.Join(".", config.UploadsDir)
logger.Infof("Upload path: %s", uploadPath)
listFiles(uploadPath)
err = http.ListenAndServe(address, router)
if err != nil {
logger.Fatalf("Server error: %s", err)
}
}
func loadConfig(filename string) error {
// Get the absolute path of the main.go file
_, currentFile, _, _ := runtime.Caller(1)
currentDir := filepath.Dir(currentFile)
// Construct the absolute path for the config file
configPath := filepath.Join(currentDir, filename)
// Open the config file
file, err := os.Open(configPath)
if err != nil {
return fmt.Errorf("error opening config file: %s", err)
}
defer file.Close()
// Decode the config file into the config variable
decoder := json.NewDecoder(file)
err = decoder.Decode(&config)
if err != nil {
return fmt.Errorf("error decoding config file: %s", err)
}
// Set the relative path for the uploads directory
config.UploadsDir = filepath.Join(currentDir, filepath.FromSlash(config.UploadsDir))
// Create the uploads directory if it doesn't exist
if _, err := os.Stat(config.UploadsDir); os.IsNotExist(err) {
err := os.MkdirAll(config.UploadsDir, os.ModePerm)
if err != nil {
return fmt.Errorf("error creating uploads directory: %s", err)
}
} }
*d = Duration(parsedDuration)
return nil return nil
} }
func serveCDN(w http.ResponseWriter, r *http.Request) { // CDNHandler handles HTTP requests to the CDN server
// Log the request information type CDNHandler struct {
logger.Infof("Received request: %s %s", r.Method, r.URL.Path) Config Config
Logger *logrus.Logger
// Check if the request has valid authentication
if !checkAuthentication(r) {
logger.Warn("Authentication failed")
w.Header().Set("WWW-Authenticate", `Basic realm="CDN Authentication"`)
w.WriteHeader(http.StatusUnauthorized)
fmt.Fprintf(w, "401 Unauthorized\n")
return
}
// Get the requested file path
filePath := filepath.Join(config.UploadsDir, filepath.Clean(r.URL.Path))
// Check if the file exists
_, err := os.Stat(filePath)
if err != nil {
if os.IsNotExist(err) {
logger.Infof("File not found: %s", r.URL.Path)
http.NotFound(w, r)
} else {
logger.Error("Internal Server Error:", err)
http.Error(w, "Internal Server Error", http.StatusInternalServerError)
}
return
}
// Serve the file
logger.Infof("Serving file: %s", r.URL.Path)
http.ServeFile(w, r, filePath)
} }
func handleUpload(w http.ResponseWriter, r *http.Request) { // ServeHTTP serves HTTP requests
// Log the request information func (c *CDNHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
logger.Infof("Received upload request: %s %s", r.Method, r.URL.Path) switch r.Method {
case http.MethodGet:
// Check if the request has valid authentication c.handleGet(w, r)
if !checkAuthentication(r) { case http.MethodPost:
logger.Warn("Authentication failed") c.handlePost(w, r)
w.Header().Set("WWW-Authenticate", `Basic realm="CDN Authentication"`) default:
w.WriteHeader(http.StatusUnauthorized) http.Error(w, "Method Not Allowed", http.StatusMethodNotAllowed)
fmt.Fprintf(w, "401 Unauthorized\n")
return
} }
}
// Parse the uploaded file // handleGet handles GET requests
err := r.ParseMultipartForm(32 << 20) // Max file size: 32MB func (c *CDNHandler) handleGet(w http.ResponseWriter, r *http.Request) {
if err != nil { c.Logger.Infof("Received GET request for URL: %s", r.URL.Path)
logger.Error("Bad Request:", err)
http.Error(w, "Bad Request", http.StatusBadRequest)
return
}
file, handler, err := r.FormFile("file") // Serve file for download
filePath := filepath.Join(c.Config.UploadDirectory, r.URL.Path)
file, err := os.Open(filePath)
if err != nil { if err != nil {
logger.Error("No file provided in the request:", err) c.Logger.Errorf("Error opening file: %v", err)
http.Error(w, "No file provided in the request", http.StatusBadRequest) http.Error(w, "File Not Found", http.StatusNotFound)
return return
} }
defer file.Close() defer file.Close()
// Validate filename // Set the appropriate Content-Type header based on file extension
filename := sanitizeFilename(handler.Filename) contentType := "application/octet-stream"
if !isValidFilename(filename) { switch filepath.Ext(filePath) {
logger.Errorf("Invalid filename: %s", handler.Filename) case ".jpg", ".jpeg":
http.Error(w, "Invalid filename", http.StatusBadRequest) contentType = "image/jpeg"
return case ".png":
contentType = "image/png"
case ".pdf":
contentType = "application/pdf"
} }
// Create the uploads directory if it doesn't exist w.Header().Set("Content-Type", contentType)
err = os.MkdirAll(config.UploadsDir, os.ModePerm) if _, err := io.Copy(w, file); err != nil {
if err != nil { c.Logger.Errorf("Error copying file: %v", err)
logger.Error("Error creating uploads directory:", err)
http.Error(w, "Error creating uploads directory", http.StatusInternalServerError)
return
}
// Create a new file in the uploads directory
dst, err := os.Create(filepath.Join(config.UploadsDir, filename))
if err != nil {
logger.Error("Internal Server Error:", err)
http.Error(w, "Internal Server Error", http.StatusInternalServerError)
return
}
defer dst.Close()
// Copy the uploaded file to the destination
_, err = io.Copy(dst, file)
if err != nil {
logger.Error("Internal Server Error:", err)
http.Error(w, "Internal Server Error", http.StatusInternalServerError) http.Error(w, "Internal Server Error", http.StatusInternalServerError)
return return
} }
logger.Infof("File uploaded successfully: %s", filename) c.Logger.Infof("File downloaded successfully: %s", r.URL.Path)
fmt.Fprintf(w, "File uploaded successfully!")
}
func checkAuthentication(r *http.Request) bool {
username, password, ok := r.BasicAuth()
return ok && username == config.Username && password == config.Password
} }
func sanitizeFilename(filename string) string { // handlePost handles POST requests
return strings.TrimSpace(filename) func (c *CDNHandler) handlePost(w http.ResponseWriter, r *http.Request) {
c.Logger.Info("Received POST request")
// Validate request size
r.Body = http.MaxBytesReader(w, r.Body, c.Config.MaxUploadSize)
if err := r.ParseMultipartForm(c.Config.MaxUploadSize); err != nil {
c.Logger.Errorf("Error parsing multipart form: %v", err)
http.Error(w, "Payload Too Large", http.StatusRequestEntityTooLarge)
return
}
// Get the uploaded file
file, handler, err := r.FormFile("file")
if err != nil {
c.Logger.Errorf("Error retrieving file: %v", err)
http.Error(w, "Error retrieving file", http.StatusInternalServerError)
return
}
defer file.Close()
// Create the upload directory if it doesn't exist
uploadDir := c.Config.UploadDirectory
if uploadDir == "" {
uploadDir = "uploads"
}
if err := os.MkdirAll(uploadDir, os.ModePerm); err != nil {
c.Logger.Errorf("Error creating upload directory: %v", err)
http.Error(w, "Error creating upload directory", http.StatusInternalServerError)
return
}
// Create the file in the upload directory
filePath := filepath.Join(uploadDir, handler.Filename)
newFile, err := os.Create(filePath)
if err != nil {
c.Logger.Errorf("Error creating file: %v", err)
http.Error(w, "Error creating file", http.StatusInternalServerError)
return
}
defer newFile.Close()
// Copy the uploaded file to the new file
if _, err := io.Copy(newFile, file); err != nil {
c.Logger.Errorf("Error copying file: %v", err)
http.Error(w, "Error copying file", http.StatusInternalServerError)
return
}
c.Logger.Infof("File uploaded successfully: %s", handler.Filename)
fmt.Fprint(w, "File uploaded successfully")
} }
func isValidFilename(filename string) bool { func main() {
return filenameRegex.MatchString(filename) // Parse command line flags
} configPath := flag.String("config", "config.json", "Path to the configuration file")
flag.Parse()
func listFiles(dirPath string) { // Initialize logrus logger
logger.Infof("Files in %s:", dirPath) logger := logrus.New()
logger.SetFormatter(&logrus.TextFormatter{
err := filepath.Walk(dirPath, func(path string, info os.FileInfo, err error) error { FullTimestamp: true,
if err != nil { TimestampFormat: "2006-01-02 15:04:05",
logger.Errorf("Error accessing file: %s", err)
return nil
}
if !info.IsDir() {
logger.Infof("- %s", path)
}
return nil
}) })
// Read the configuration file
configFile, err := os.Open(*configPath)
if err != nil { if err != nil {
logger.Errorf("Error listing files: %s", err) logger.Fatalf("Error opening configuration file: %v", err)
} }
defer configFile.Close()
// Decode the configuration file
var config Config
err = json.NewDecoder(configFile).Decode(&config)
if err != nil {
logger.Fatalf("Error decoding configuration file: %v", err)
}
// Start a goroutine to restart the server periodically
if config.Heartbeat > 0 {
go func() {
for range time.Tick(time.Duration(config.Heartbeat)) {
logger.Info("Server heartbeat")
os.Exit(0)
}
}()
}
// Create a new CDNHandler with the configuration
cdnHandler := &CDNHandler{
Config: config,
Logger: logger,
}
// Create a new HTTP server
server := &http.Server{
Addr: ":" + config.Port,
Handler: cdnHandler,
ErrorLog: log.New(logger.Writer(), "", 0),
ReadTimeout: 5 * time.Second,
WriteTimeout: 10 * time.Second,
}
logger.Infof("Starting CDN server on port %s", config.Port)
log.Fatal(server.ListenAndServe())
} }