Merge pull request #1 from NotAShelf/v2

rewrite the CDN to be more verbose and flexible
2024-11-22 16:00:43 +00:00 · 2023-06-03 22:24:21 +03:00 · 2023-06-03 22:24:21 +03:00 · b249bfd141
commit b249bfd141
parent 9a5d191c46 858e01fdc5
5 changed files with 256 additions and 240 deletions
--- a/README.md
+++ b/README.md
@ -1,12 +1,23 @@
 # go-cdn

-An experimental CDN project in Go.
+> An experimental CDN project in Go.
+> A lightweight server that allows you to upload and download files via HTTP. It provides optional authentication, input validation, error handling, and verbose logging. The server is implemented in Go and can be easily configured using a config.json file.

 ## Features

 - Serve static files securely over HTTP
 - Basic authentication support
- File upload functionality
+- File upload & download functionality
+
+### Error Handling
+
+The server is equipped with error handling to handle various scenarios, including invalid requests, authentication failures, and payload size exceeding the maximum allowed limit. If an error occurs, the server will return an appropriate HTTP status code along with an error message.
+Logging
+
+The server uses the logrus library for logging. Verbose logging is enabled by default and displays detailed information about incoming requests, errors, and server restarts. The logs are printed to the console.
+Customization
+
+The server implementation provided is a starting point and may require customization based on your specific requirements. You can modify the handleGet and handlePost functions in the CDNHandler struct to implement your desired file upload and download logic.

 ## Possible use cases

@ -18,64 +29,107 @@ The CDN can be used in various scenarios, such as:

 ## Usage

-### Starting the CDN
+### Prerequisites

-1. Build the program
+- Go programming language (version 1.16 or later)

-```go
-go build
-```
+### Installation

-2. Run the built binary
-
-```go
-
-```
-
-### Using the CDN
-
-To request a file from the CDN, use the following URL format:
+1. Clone the repository:

 ```bash
-http://your-cdn-server:port/file-path
+
+git clone https://github.com/notashelf/go-cdn.git
 ```

-> Replace your-cdn-server with the hostname or IP address of your CDN server and port with the port number on which the server is running. Append the desired file path after the hostname and port.
-
-**Example:**
-
- `http://localhost:8080/images/file_you_have_uploaded.png`
-
-  - If the file exists and the request is authorized, the file will be served by the CDN.
-  - If the file doesn't exist, a 404 Not Found response will be returned.
-
-#### Uploading Files to the CDN
-
-> Send a POST request to the /upload endpoint of the CDN server.
-
-Example using cURL:
+2. Change to the project directory:

 ```bash
-curl -X POST -u username:password -F "file=@/path/to/file" http://your-cdn-server:port/upload
+cd your-repo
 ```

-> Replace username and password with the authentication credentials you have set in the main.go file. Replace your-cdn-server with the hostname or IP address of your CDN server and port with the port number on which the server is running. Provide the file path after the @ symbol in the -F parameter.
-
-**Example:**
+3. Install the dependencies:

 ```bash
-curl -X POST -u admin:password -F "file=@/absolute/path/to/image.jpg" http://localhost:8080/upload
+go get github.com/sirupsen/logrus
+go get github.com/pkg/errors
 ```

-    - The uploaded file will be saved in the specified uploadPath directory.
+### Configuration

-> Note: The server responds with a success message if the file upload is successful.
+The server can be configured using the config.json file. Create a file named config.json in the same directory as the main file (main.go) and configure the following properties:

-### Security Considerations
+- port (string): The port on which the server will listen for incoming connections.
+- max_upload_size (integer): The maximum allowed size of file uploads, in bytes.
+- heartbeat (string): The duration after which the server will automatically restart. Specify a value in the format "5m" for 5 minutes, "1h" for 1 hour, etc. Set to "0" to disable automatic restarts.
+- require_auth (boolean): Whether to require authentication for file uploads and downloads.
+- auth_username (string): The username for authentication (only applicable if require_auth is set to true).
+- auth_password (string): The password for authentication (only applicable if require_auth is set to true).

-It is highly recommended to use SSL/TLS encryption (HTTPS) for secure communication between clients and the CDN server.
-Change the default username and password in the main.go file to strong and secure credentials.
-Consider implementing additional security measures based on your specific requirements.
+Example config.json file:
+
+```json
+{
+  "port": "8080",
+  "max_upload_size": 10485760,
+  "heartbeat": "1h",
+  "require_auth": true,
+  "auth_username": "your-username",
+  "auth_password": "your-password"
+}
+```
+
+### Usage
+
+Start the CDN server by running the following command:
+
+```bash
+go run main.go -config config.json
+```
+
+The server will start and listen on the specified port. Verbose log messages will be displayed in the console.
+
+#### Uploading a file:
+
+Use the curl command to upload a file to the CDN server:
+
+```bash
+curl -X POST -F "file=@/path/to/file" http://localhost:8080
+```
+
+_Replace /path/to/file with the actual path to the file you want to upload. If authentication is enabled, provide the username and password when prompted._
+
+#### Downloading a file:
+
+Use the curl command to download a file from the CDN server:
+
+```bash
+curl -O http://localhost:8080/<filename>
+```
+
+_Replace <filename> with the name of the file you want to download._
+
+### Authenticated Upload and Download
+
+To perform an authenticated upload or download, you can use the following curl commands:
+
+#### Uploading a File:
+
+```bash
+curl -X POST -F "file=@/path/to/file" -u "your-username:your-password" http://localhost:8080
+```
+
+_Replace /path/to/file with the actual path to the file you want to upload. The -u flag is used to provide the authentication credentials._
+
+#### Downloading a File:
+
+```bash
+curl -O -u "your-username:your-password" http://localhost:8080/<filename>
+```
+
+Replace <filename> with the name of the file you want to download. The -u flag is used to provide the authentication credentials.
+
+**Please note that these examples assume you're running the server on localhost with the specified port and authentication credentials. Make sure to adjust the hostname and port accordingly.**

 # License

--- a/example_config.json
+++ b/example_config.json
@ -1,6 +0,0 @@
-{
-  "username": "admin",
-  "password": "password",
-  "service_port": "8080",
-  "uploads_dir": "uploads/"
-}
--- a/go.mod
+++ b/go.mod
@ -3,7 +3,7 @@ module cdn
 go 1.20

 require (
-	github.com/go-chi/chi v1.5.4 // indirect
+	github.com/pkg/errors v0.9.1 // indirect
 	github.com/sirupsen/logrus v1.9.2 // indirect
-	golang.org/x/sys v0.8.0 // indirect
+	golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8 // indirect
 )
--- a/go.sum
+++ b/go.sum
@ -1,14 +1,13 @@
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
-github.com/go-chi/chi v1.5.4 h1:QHdzF2szwjqVV4wmByUnTcsbIg7UGaQ0tPF2t5GcAIs=
-github.com/go-chi/chi v1.5.4/go.mod h1:uaf8YgoFazUOkPBG7fxPftUylNumIev9awIWOENIuEg=
+github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
+github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
 github.com/sirupsen/logrus v1.9.2 h1:oxx1eChJGI6Uks2ZC4W1zpLlVgqB8ner4EuQwV4Ik1Y=
 github.com/sirupsen/logrus v1.9.2/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
 github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
 github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
+golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8 h1:0A+M6Uqn+Eje4kHMK80dtF3JCXC4ykBgQG4Fe06QRhQ=
 golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.8.0 h1:EBmGv8NaZBZTWvrbjNoL6HVt+IVy3QDQpJs7VRIw3tU=
-golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
--- a/main.go
+++ b/main.go
@ -2,236 +2,205 @@ package main

 import (
 	"encoding/json"
+	"flag"
 	"fmt"
 	"io"
+	"log"
 	"net/http"
 	"os"
 	"path/filepath"
-	"regexp"
-	"runtime"
-	"strings"
+	"time"

-	"github.com/go-chi/chi"
-	"github.com/go-chi/chi/middleware"
 	"github.com/sirupsen/logrus"
 )

+// Config represents the configuration structure
 type Config struct {
-	Username    string `json:"username"`
-	Password    string `json:"password"`
-	ServicePort string `json:"service_port"`
-	UploadsDir  string `json:"uploads_dir"`
+	Port            string   `json:"port"`
+	MaxUploadSize   int64    `json:"max_upload_size"`
+	Heartbeat       Duration `json:"heartbeat"`
+	RequireAuth     bool     `json:"require_auth"`
+	AuthUsername    string   `json:"auth_username"`
+	AuthPassword    string   `json:"auth_password"`
+	UploadDirectory string   `json:"upload_directory"`
 }

-var (
-	config        Config
-	logger        *logrus.Logger
-	filenameRegex *regexp.Regexp
-)
+// Duration is a custom type for decoding time.Duration from JSON
+type Duration time.Duration

-func main() {
-	// Initialize logger
-	logger = logrus.New()
-	logger.Formatter = &logrus.TextFormatter{
-		DisableTimestamp: false,
-		FullTimestamp:    true,
+// UnmarshalJSON unmarshals a JSON string into a Duration
+func (d *Duration) UnmarshalJSON(data []byte) error {
+	var durationStr string
+	if err := json.Unmarshal(data, &durationStr); err != nil {
+		return fmt.Errorf("error decoding Duration: %w", err)
 	}

-	// Load configuration
-	err := loadConfig("config.json")
+	parsedDuration, err := time.ParseDuration(durationStr)
 	if err != nil {
-		logger.Fatalf("Error loading config file: %s", err)
-	}
-
-	// Initialize filename regular expression
-	filenameRegex = regexp.MustCompile(`^[a-zA-Z0-9-_\.]+$`)
-
-	// Initialize router
-	router := chi.NewRouter()
-	router.Use(middleware.Logger)
-
-	router.HandleFunc("/", serveCDN)
-	router.HandleFunc("/upload", handleUpload)
-
-	// Start server
-	address := fmt.Sprintf(":%s", config.ServicePort)
-	logger.Infof("Starting CDN server on port %s...", config.ServicePort)
-	logger.Infof("Serving files from %s", config.UploadsDir)
-
-	// Print upload path and list files
-	uploadPath := filepath.Join(".", config.UploadsDir)
-	logger.Infof("Upload path: %s", uploadPath)
-	listFiles(uploadPath)
-
-	err = http.ListenAndServe(address, router)
-	if err != nil {
-		logger.Fatalf("Server error: %s", err)
-	}
-}
-
-func loadConfig(filename string) error {
-	// Get the absolute path of the main.go file
-	_, currentFile, _, _ := runtime.Caller(1)
-	currentDir := filepath.Dir(currentFile)
-
-	// Construct the absolute path for the config file
-	configPath := filepath.Join(currentDir, filename)
-
-	// Open the config file
-	file, err := os.Open(configPath)
-	if err != nil {
-		return fmt.Errorf("error opening config file: %s", err)
-	}
-	defer file.Close()
-
-	// Decode the config file into the config variable
-	decoder := json.NewDecoder(file)
-	err = decoder.Decode(&config)
-	if err != nil {
-		return fmt.Errorf("error decoding config file: %s", err)
-	}
-
-	// Set the relative path for the uploads directory
-	config.UploadsDir = filepath.Join(currentDir, filepath.FromSlash(config.UploadsDir))
-
-	// Create the uploads directory if it doesn't exist
-	if _, err := os.Stat(config.UploadsDir); os.IsNotExist(err) {
-		err := os.MkdirAll(config.UploadsDir, os.ModePerm)
-		if err != nil {
-			return fmt.Errorf("error creating uploads directory: %s", err)
-		}
+		return fmt.Errorf("error parsing Duration: %w", err)
 	}

+	*d = Duration(parsedDuration)
 	return nil
 }

-func serveCDN(w http.ResponseWriter, r *http.Request) {
-	// Log the request information
-	logger.Infof("Received request: %s %s", r.Method, r.URL.Path)
-
-	// Check if the request has valid authentication
-	if !checkAuthentication(r) {
-		logger.Warn("Authentication failed")
-		w.Header().Set("WWW-Authenticate", `Basic realm="CDN Authentication"`)
-		w.WriteHeader(http.StatusUnauthorized)
-		fmt.Fprintf(w, "401 Unauthorized\n")
-		return
-	}
-
-	// Get the requested file path
-	filePath := filepath.Join(config.UploadsDir, filepath.Clean(r.URL.Path))
-
-	// Check if the file exists
-	_, err := os.Stat(filePath)
-	if err != nil {
-		if os.IsNotExist(err) {
-			logger.Infof("File not found: %s", r.URL.Path)
-			http.NotFound(w, r)
-		} else {
-			logger.Error("Internal Server Error:", err)
-			http.Error(w, "Internal Server Error", http.StatusInternalServerError)
-		}
-		return
-	}
-
-	// Serve the file
-	logger.Infof("Serving file: %s", r.URL.Path)
-	http.ServeFile(w, r, filePath)
+// CDNHandler handles HTTP requests to the CDN server
+type CDNHandler struct {
+	Config Config
+	Logger *logrus.Logger
 }

-func handleUpload(w http.ResponseWriter, r *http.Request) {
-	// Log the request information
-	logger.Infof("Received upload request: %s %s", r.Method, r.URL.Path)
-
-	// Check if the request has valid authentication
-	if !checkAuthentication(r) {
-		logger.Warn("Authentication failed")
-		w.Header().Set("WWW-Authenticate", `Basic realm="CDN Authentication"`)
-		w.WriteHeader(http.StatusUnauthorized)
-		fmt.Fprintf(w, "401 Unauthorized\n")
-		return
+// ServeHTTP serves HTTP requests
+func (c *CDNHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
+	switch r.Method {
+	case http.MethodGet:
+		c.handleGet(w, r)
+	case http.MethodPost:
+		c.handlePost(w, r)
+	default:
+		http.Error(w, "Method Not Allowed", http.StatusMethodNotAllowed)
 	}
+}

-	// Parse the uploaded file
-	err := r.ParseMultipartForm(32 << 20) // Max file size: 32MB
-	if err != nil {
-		logger.Error("Bad Request:", err)
-		http.Error(w, "Bad Request", http.StatusBadRequest)
-		return
-	}
+// handleGet handles GET requests
+func (c *CDNHandler) handleGet(w http.ResponseWriter, r *http.Request) {
+	c.Logger.Infof("Received GET request for URL: %s", r.URL.Path)

-	file, handler, err := r.FormFile("file")
+	// Serve file for download
+	filePath := filepath.Join(c.Config.UploadDirectory, r.URL.Path)
+	file, err := os.Open(filePath)
 	if err != nil {
-		logger.Error("No file provided in the request:", err)
-		http.Error(w, "No file provided in the request", http.StatusBadRequest)
+		c.Logger.Errorf("Error opening file: %v", err)
+		http.Error(w, "File Not Found", http.StatusNotFound)
 		return
 	}
 	defer file.Close()

-	// Validate filename
-	filename := sanitizeFilename(handler.Filename)
-	if !isValidFilename(filename) {
-		logger.Errorf("Invalid filename: %s", handler.Filename)
-		http.Error(w, "Invalid filename", http.StatusBadRequest)
-		return
+	// Set the appropriate Content-Type header based on file extension
+	contentType := "application/octet-stream"
+	switch filepath.Ext(filePath) {
+	case ".jpg", ".jpeg":
+		contentType = "image/jpeg"
+	case ".png":
+		contentType = "image/png"
+	case ".pdf":
+		contentType = "application/pdf"
 	}

-	// Create the uploads directory if it doesn't exist
-	err = os.MkdirAll(config.UploadsDir, os.ModePerm)
-	if err != nil {
-		logger.Error("Error creating uploads directory:", err)
-		http.Error(w, "Error creating uploads directory", http.StatusInternalServerError)
-		return
-	}
-
-	// Create a new file in the uploads directory
-	dst, err := os.Create(filepath.Join(config.UploadsDir, filename))
-	if err != nil {
-		logger.Error("Internal Server Error:", err)
-		http.Error(w, "Internal Server Error", http.StatusInternalServerError)
-		return
-	}
-	defer dst.Close()
-
-	// Copy the uploaded file to the destination
-	_, err = io.Copy(dst, file)
-	if err != nil {
-		logger.Error("Internal Server Error:", err)
+	w.Header().Set("Content-Type", contentType)
+	if _, err := io.Copy(w, file); err != nil {
+		c.Logger.Errorf("Error copying file: %v", err)
 		http.Error(w, "Internal Server Error", http.StatusInternalServerError)
 		return
 	}

-	logger.Infof("File uploaded successfully: %s", filename)
-	fmt.Fprintf(w, "File uploaded successfully!")
-}
-func checkAuthentication(r *http.Request) bool {
-	username, password, ok := r.BasicAuth()
-	return ok && username == config.Username && password == config.Password
+	c.Logger.Infof("File downloaded successfully: %s", r.URL.Path)
 }

-func sanitizeFilename(filename string) string {
-	return strings.TrimSpace(filename)
+// handlePost handles POST requests
+func (c *CDNHandler) handlePost(w http.ResponseWriter, r *http.Request) {
+	c.Logger.Info("Received POST request")
+
+	// Validate request size
+	r.Body = http.MaxBytesReader(w, r.Body, c.Config.MaxUploadSize)
+	if err := r.ParseMultipartForm(c.Config.MaxUploadSize); err != nil {
+		c.Logger.Errorf("Error parsing multipart form: %v", err)
+		http.Error(w, "Payload Too Large", http.StatusRequestEntityTooLarge)
+		return
+	}
+
+	// Get the uploaded file
+	file, handler, err := r.FormFile("file")
+	if err != nil {
+		c.Logger.Errorf("Error retrieving file: %v", err)
+		http.Error(w, "Error retrieving file", http.StatusInternalServerError)
+		return
+	}
+	defer file.Close()
+
+	// Create the upload directory if it doesn't exist
+	uploadDir := c.Config.UploadDirectory
+	if uploadDir == "" {
+		uploadDir = "uploads"
+	}
+	if err := os.MkdirAll(uploadDir, os.ModePerm); err != nil {
+		c.Logger.Errorf("Error creating upload directory: %v", err)
+		http.Error(w, "Error creating upload directory", http.StatusInternalServerError)
+		return
+	}
+
+	// Create the file in the upload directory
+	filePath := filepath.Join(uploadDir, handler.Filename)
+	newFile, err := os.Create(filePath)
+	if err != nil {
+		c.Logger.Errorf("Error creating file: %v", err)
+		http.Error(w, "Error creating file", http.StatusInternalServerError)
+		return
+	}
+	defer newFile.Close()
+
+	// Copy the uploaded file to the new file
+	if _, err := io.Copy(newFile, file); err != nil {
+		c.Logger.Errorf("Error copying file: %v", err)
+		http.Error(w, "Error copying file", http.StatusInternalServerError)
+		return
+	}
+
+	c.Logger.Infof("File uploaded successfully: %s", handler.Filename)
+	fmt.Fprint(w, "File uploaded successfully")
 }

-func isValidFilename(filename string) bool {
-	return filenameRegex.MatchString(filename)
-}
+func main() {
+	// Parse command line flags
+	configPath := flag.String("config", "config.json", "Path to the configuration file")
+	flag.Parse()

-func listFiles(dirPath string) {
-	logger.Infof("Files in %s:", dirPath)
-
-	err := filepath.Walk(dirPath, func(path string, info os.FileInfo, err error) error {
-		if err != nil {
-			logger.Errorf("Error accessing file: %s", err)
-			return nil
-		}
-		if !info.IsDir() {
-			logger.Infof("- %s", path)
-		}
-		return nil
+	// Initialize logrus logger
+	logger := logrus.New()
+	logger.SetFormatter(&logrus.TextFormatter{
+		FullTimestamp:   true,
+		TimestampFormat: "2006-01-02 15:04:05",
 	})

+	// Read the configuration file
+	configFile, err := os.Open(*configPath)
 	if err != nil {
-		logger.Errorf("Error listing files: %s", err)
+		logger.Fatalf("Error opening configuration file: %v", err)
 	}
+	defer configFile.Close()
+
+	// Decode the configuration file
+	var config Config
+	err = json.NewDecoder(configFile).Decode(&config)
+	if err != nil {
+		logger.Fatalf("Error decoding configuration file: %v", err)
+	}
+
+	// Start a goroutine to restart the server periodically
+	if config.Heartbeat > 0 {
+		go func() {
+			for range time.Tick(time.Duration(config.Heartbeat)) {
+				logger.Info("Server heartbeat")
+				os.Exit(0)
+			}
+		}()
+	}
+
+	// Create a new CDNHandler with the configuration
+	cdnHandler := &CDNHandler{
+		Config: config,
+		Logger: logger,
+	}
+
+	// Create a new HTTP server
+	server := &http.Server{
+		Addr:         ":" + config.Port,
+		Handler:      cdnHandler,
+		ErrorLog:     log.New(logger.Writer(), "", 0),
+		ReadTimeout:  5 * time.Second,
+		WriteTimeout: 10 * time.Second,
+	}
+
+	logger.Infof("Starting CDN server on port %s", config.Port)
+	log.Fatal(server.ListenAndServe())
 }