NotAShelf/go-cdn
1
0
Fork 0
mirror of https://github.com/NotAShelf/go-cdn.git synced 2024-11-22 16:00:43 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Merge pull request #1 from NotAShelf/v2

Browse source 
rewrite the CDN to be more verbose and flexible
This commit is contained in:
NotAShelf 2023-06-03 22:24:21 +03:00 committed by GitHub
commit b249bfd141
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
5 changed files with 256 additions and 240 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

138
README.md
View file

@ -1,12 +1,23 @@
# go-cdn
An experimental CDN project in Go.
> An experimental CDN project in Go.
> A lightweight server that allows you to upload and download files via HTTP. It provides optional authentication, input validation, error handling, and verbose logging. The server is implemented in Go and can be easily configured using a config.json file.
## Features
- Serve static files securely over HTTP
- Basic authentication support
- File upload functionality
- File upload & download functionality
### Error Handling
The server is equipped with error handling to handle various scenarios, including invalid requests, authentication failures, and payload size exceeding the maximum allowed limit. If an error occurs, the server will return an appropriate HTTP status code along with an error message.
Logging
The server uses the logrus library for logging. Verbose logging is enabled by default and displays detailed information about incoming requests, errors, and server restarts. The logs are printed to the console.
Customization
The server implementation provided is a starting point and may require customization based on your specific requirements. You can modify the handleGet and handlePost functions in the CDNHandler struct to implement your desired file upload and download logic.
## Possible use cases
@ -18,64 +29,107 @@ The CDN can be used in various scenarios, such as:
## Usage
### Starting the CDN
### Prerequisites
1. Build the program
- Go programming language (version 1.16 or later)
```go
go build
```
### Installation
2. Run the built binary
```go
```
### Using the CDN
To request a file from the CDN, use the following URL format:
1. Clone the repository:
```bash
http://your-cdn-server:port/file-path
git clone https://github.com/notashelf/go-cdn.git
```
> Replace your-cdn-server with the hostname or IP address of your CDN server and port with the port number on which the server is running. Append the desired file path after the hostname and port.
**Example:**
- `http://localhost:8080/images/file_you_have_uploaded.png`
- If the file exists and the request is authorized, the file will be served by the CDN.
- If the file doesn't exist, a 404 Not Found response will be returned.
#### Uploading Files to the CDN
> Send a POST request to the /upload endpoint of the CDN server.
Example using cURL:
2. Change to the project directory:
```bash
curl -X POST -u username:password -F "file=@/path/to/file" http://your-cdn-server:port/upload
cd your-repo
```
> Replace username and password with the authentication credentials you have set in the main.go file. Replace your-cdn-server with the hostname or IP address of your CDN server and port with the port number on which the server is running. Provide the file path after the @ symbol in the -F parameter.
**Example:**
3. Install the dependencies:
```bash
curl -X POST -u admin:password -F "file=@/absolute/path/to/image.jpg" http://localhost:8080/upload
go get github.com/sirupsen/logrus
go get github.com/pkg/errors
```
- The uploaded file will be saved in the specified uploadPath directory.
### Configuration
> Note: The server responds with a success message if the file upload is successful.
The server can be configured using the config.json file. Create a file named config.json in the same directory as the main file (main.go) and configure the following properties:
### Security Considerations
- port (string): The port on which the server will listen for incoming connections.
- max_upload_size (integer): The maximum allowed size of file uploads, in bytes.
- heartbeat (string): The duration after which the server will automatically restart. Specify a value in the format "5m" for 5 minutes, "1h" for 1 hour, etc. Set to "0" to disable automatic restarts.
- require_auth (boolean): Whether to require authentication for file uploads and downloads.
- auth_username (string): The username for authentication (only applicable if require_auth is set to true).
- auth_password (string): The password for authentication (only applicable if require_auth is set to true).
It is highly recommended to use SSL/TLS encryption (HTTPS) for secure communication between clients and the CDN server.
Change the default username and password in the main.go file to strong and secure credentials.
Consider implementing additional security measures based on your specific requirements.
Example config.json file:
```json
{
"port": "8080",
"max_upload_size": 10485760,
"heartbeat": "1h",
"require_auth": true,
"auth_username": "your-username",
"auth_password": "your-password"
}
```
### Usage
Start the CDN server by running the following command:
```bash
go run main.go -config config.json
```
The server will start and listen on the specified port. Verbose log messages will be displayed in the console.
#### Uploading a file:
Use the curl command to upload a file to the CDN server:
```bash
curl -X POST -F "file=@/path/to/file" http://localhost:8080
```
_Replace /path/to/file with the actual path to the file you want to upload. If authentication is enabled, provide the username and password when prompted._
#### Downloading a file:
Use the curl command to download a file from the CDN server:
```bash
curl -O http://localhost:8080/<filename>
```
_Replace <filename> with the name of the file you want to download._
### Authenticated Upload and Download
To perform an authenticated upload or download, you can use the following curl commands:
#### Uploading a File:
```bash
curl -X POST -F "file=@/path/to/file" -u "your-username:your-password" http://localhost:8080
```
_Replace /path/to/file with the actual path to the file you want to upload. The -u flag is used to provide the authentication credentials._
#### Downloading a File:
```bash
curl -O -u "your-username:your-password" http://localhost:8080/<filename>
```
Replace <filename> with the name of the file you want to download. The -u flag is used to provide the authentication credentials.
**Please note that these examples assume you're running the server on localhost with the specified port and authentication credentials. Make sure to adjust the hostname and port accordingly.**
# License

6
example_config.json
View file

@ -1,6 +0,0 @@
{
"username": "admin",
"password": "password",
"service_port": "8080",
"uploads_dir": "uploads/"
}

4
go.mod
View file

@ -3,7 +3,7 @@ module cdn
go 1.20
require (
github.com/go-chi/chi v1.5.4 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/sirupsen/logrus v1.9.2 // indirect
golang.org/x/sys v0.8.0 // indirect
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8 // indirect
)

7
go.sum
View file

@ -1,14 +1,13 @@
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/go-chi/chi v1.5.4 h1:QHdzF2szwjqVV4wmByUnTcsbIg7UGaQ0tPF2t5GcAIs=
github.com/go-chi/chi v1.5.4/go.mod h1:uaf8YgoFazUOkPBG7fxPftUylNumIev9awIWOENIuEg=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/sirupsen/logrus v1.9.2 h1:oxx1eChJGI6Uks2ZC4W1zpLlVgqB8ner4EuQwV4Ik1Y=
github.com/sirupsen/logrus v1.9.2/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8 h1:0A+M6Uqn+Eje4kHMK80dtF3JCXC4ykBgQG4Fe06QRhQ=
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.8.0 h1:EBmGv8NaZBZTWvrbjNoL6HVt+IVy3QDQpJs7VRIw3tU=
golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

341
main.go
View file

@ -2,236 +2,205 @@ package main
import (
"encoding/json"
"flag"
"fmt"
"io"
"log"
"net/http"
"os"
"path/filepath"
"regexp"
"runtime"
"strings"
"time"
"github.com/go-chi/chi"
"github.com/go-chi/chi/middleware"
"github.com/sirupsen/logrus"
)
// Config represents the configuration structure
type Config struct {
Username string `json:"username"`
Password string `json:"password"`
ServicePort string `json:"service_port"`
UploadsDir string `json:"uploads_dir"`
Port string `json:"port"`
MaxUploadSize int64 `json:"max_upload_size"`
Heartbeat Duration `json:"heartbeat"`
RequireAuth bool `json:"require_auth"`
AuthUsername string `json:"auth_username"`
AuthPassword string `json:"auth_password"`
UploadDirectory string `json:"upload_directory"`
}
var (
config Config
logger *logrus.Logger
filenameRegex *regexp.Regexp
)
// Duration is a custom type for decoding time.Duration from JSON
type Duration time.Duration
func main() {
// Initialize logger
logger = logrus.New()
logger.Formatter = &logrus.TextFormatter{
DisableTimestamp: false,
FullTimestamp: true,
// UnmarshalJSON unmarshals a JSON string into a Duration
func (d *Duration) UnmarshalJSON(data []byte) error {
var durationStr string
if err := json.Unmarshal(data, &durationStr); err != nil {
return fmt.Errorf("error decoding Duration: %w", err)
}
// Load configuration
err := loadConfig("config.json")
parsedDuration, err := time.ParseDuration(durationStr)
if err != nil {
logger.Fatalf("Error loading config file: %s", err)
}
// Initialize filename regular expression
filenameRegex = regexp.MustCompile(`^[a-zA-Z0-9-_\.]+$`)
// Initialize router
router := chi.NewRouter()
router.Use(middleware.Logger)
router.HandleFunc("/", serveCDN)
router.HandleFunc("/upload", handleUpload)
// Start server
address := fmt.Sprintf(":%s", config.ServicePort)
logger.Infof("Starting CDN server on port %s...", config.ServicePort)
logger.Infof("Serving files from %s", config.UploadsDir)
// Print upload path and list files
uploadPath := filepath.Join(".", config.UploadsDir)
logger.Infof("Upload path: %s", uploadPath)
listFiles(uploadPath)
err = http.ListenAndServe(address, router)
if err != nil {
logger.Fatalf("Server error: %s", err)
}
}
func loadConfig(filename string) error {
// Get the absolute path of the main.go file
_, currentFile, _, _ := runtime.Caller(1)
currentDir := filepath.Dir(currentFile)
// Construct the absolute path for the config file
configPath := filepath.Join(currentDir, filename)
// Open the config file
file, err := os.Open(configPath)
if err != nil {
return fmt.Errorf("error opening config file: %s", err)
}
defer file.Close()
// Decode the config file into the config variable
decoder := json.NewDecoder(file)
err = decoder.Decode(&config)
if err != nil {
return fmt.Errorf("error decoding config file: %s", err)
}
// Set the relative path for the uploads directory
config.UploadsDir = filepath.Join(currentDir, filepath.FromSlash(config.UploadsDir))
// Create the uploads directory if it doesn't exist
if _, err := os.Stat(config.UploadsDir); os.IsNotExist(err) {
err := os.MkdirAll(config.UploadsDir, os.ModePerm)
if err != nil {
return fmt.Errorf("error creating uploads directory: %s", err)
}
return fmt.Errorf("error parsing Duration: %w", err)
}
*d = Duration(parsedDuration)
return nil
}
func serveCDN(w http.ResponseWriter, r *http.Request) {
// Log the request information
logger.Infof("Received request: %s %s", r.Method, r.URL.Path)
// Check if the request has valid authentication
if !checkAuthentication(r) {
logger.Warn("Authentication failed")
w.Header().Set("WWW-Authenticate", `Basic realm="CDN Authentication"`)
w.WriteHeader(http.StatusUnauthorized)
fmt.Fprintf(w, "401 Unauthorized\n")
return
}
// Get the requested file path
filePath := filepath.Join(config.UploadsDir, filepath.Clean(r.URL.Path))
// Check if the file exists
_, err := os.Stat(filePath)
if err != nil {
if os.IsNotExist(err) {
logger.Infof("File not found: %s", r.URL.Path)
http.NotFound(w, r)
} else {
logger.Error("Internal Server Error:", err)
http.Error(w, "Internal Server Error", http.StatusInternalServerError)
}
return
}
// Serve the file
logger.Infof("Serving file: %s", r.URL.Path)
http.ServeFile(w, r, filePath)
// CDNHandler handles HTTP requests to the CDN server
type CDNHandler struct {
Config Config
Logger *logrus.Logger
}
func handleUpload(w http.ResponseWriter, r *http.Request) {
// Log the request information
logger.Infof("Received upload request: %s %s", r.Method, r.URL.Path)
// Check if the request has valid authentication
if !checkAuthentication(r) {
logger.Warn("Authentication failed")
w.Header().Set("WWW-Authenticate", `Basic realm="CDN Authentication"`)
w.WriteHeader(http.StatusUnauthorized)
fmt.Fprintf(w, "401 Unauthorized\n")
return
// ServeHTTP serves HTTP requests
func (c *CDNHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
switch r.Method {
case http.MethodGet:
c.handleGet(w, r)
case http.MethodPost:
c.handlePost(w, r)
default:
http.Error(w, "Method Not Allowed", http.StatusMethodNotAllowed)
}
}
// Parse the uploaded file
err := r.ParseMultipartForm(32 << 20) // Max file size: 32MB
if err != nil {
logger.Error("Bad Request:", err)
http.Error(w, "Bad Request", http.StatusBadRequest)
return
}
// handleGet handles GET requests
func (c *CDNHandler) handleGet(w http.ResponseWriter, r *http.Request) {
c.Logger.Infof("Received GET request for URL: %s", r.URL.Path)
file, handler, err := r.FormFile("file")
// Serve file for download
filePath := filepath.Join(c.Config.UploadDirectory, r.URL.Path)
file, err := os.Open(filePath)
if err != nil {
logger.Error("No file provided in the request:", err)
http.Error(w, "No file provided in the request", http.StatusBadRequest)
c.Logger.Errorf("Error opening file: %v", err)
http.Error(w, "File Not Found", http.StatusNotFound)
return
}
defer file.Close()
// Validate filename
filename := sanitizeFilename(handler.Filename)
if !isValidFilename(filename) {
logger.Errorf("Invalid filename: %s", handler.Filename)
http.Error(w, "Invalid filename", http.StatusBadRequest)
return
// Set the appropriate Content-Type header based on file extension
contentType := "application/octet-stream"
switch filepath.Ext(filePath) {
case ".jpg", ".jpeg":
contentType = "image/jpeg"
case ".png":
contentType = "image/png"
case ".pdf":
contentType = "application/pdf"
}
// Create the uploads directory if it doesn't exist
err = os.MkdirAll(config.UploadsDir, os.ModePerm)
if err != nil {
logger.Error("Error creating uploads directory:", err)
http.Error(w, "Error creating uploads directory", http.StatusInternalServerError)
return
}
// Create a new file in the uploads directory
dst, err := os.Create(filepath.Join(config.UploadsDir, filename))
if err != nil {
logger.Error("Internal Server Error:", err)
http.Error(w, "Internal Server Error", http.StatusInternalServerError)
return
}
defer dst.Close()
// Copy the uploaded file to the destination
_, err = io.Copy(dst, file)
if err != nil {
logger.Error("Internal Server Error:", err)
w.Header().Set("Content-Type", contentType)
if _, err := io.Copy(w, file); err != nil {
c.Logger.Errorf("Error copying file: %v", err)
http.Error(w, "Internal Server Error", http.StatusInternalServerError)
return
}
logger.Infof("File uploaded successfully: %s", filename)
fmt.Fprintf(w, "File uploaded successfully!")
}
func checkAuthentication(r *http.Request) bool {
username, password, ok := r.BasicAuth()
return ok && username == config.Username && password == config.Password
c.Logger.Infof("File downloaded successfully: %s", r.URL.Path)
}
func sanitizeFilename(filename string) string {
return strings.TrimSpace(filename)
// handlePost handles POST requests
func (c *CDNHandler) handlePost(w http.ResponseWriter, r *http.Request) {
c.Logger.Info("Received POST request")
// Validate request size
r.Body = http.MaxBytesReader(w, r.Body, c.Config.MaxUploadSize)
if err := r.ParseMultipartForm(c.Config.MaxUploadSize); err != nil {
c.Logger.Errorf("Error parsing multipart form: %v", err)
http.Error(w, "Payload Too Large", http.StatusRequestEntityTooLarge)
return
}
// Get the uploaded file
file, handler, err := r.FormFile("file")
if err != nil {
c.Logger.Errorf("Error retrieving file: %v", err)
http.Error(w, "Error retrieving file", http.StatusInternalServerError)
return
}
defer file.Close()
// Create the upload directory if it doesn't exist
uploadDir := c.Config.UploadDirectory
if uploadDir == "" {
uploadDir = "uploads"
}
if err := os.MkdirAll(uploadDir, os.ModePerm); err != nil {
c.Logger.Errorf("Error creating upload directory: %v", err)
http.Error(w, "Error creating upload directory", http.StatusInternalServerError)
return
}
// Create the file in the upload directory
filePath := filepath.Join(uploadDir, handler.Filename)
newFile, err := os.Create(filePath)
if err != nil {
c.Logger.Errorf("Error creating file: %v", err)
http.Error(w, "Error creating file", http.StatusInternalServerError)
return
}
defer newFile.Close()
// Copy the uploaded file to the new file
if _, err := io.Copy(newFile, file); err != nil {
c.Logger.Errorf("Error copying file: %v", err)
http.Error(w, "Error copying file", http.StatusInternalServerError)
return
}
c.Logger.Infof("File uploaded successfully: %s", handler.Filename)
fmt.Fprint(w, "File uploaded successfully")
}
func isValidFilename(filename string) bool {
return filenameRegex.MatchString(filename)
}
func main() {
// Parse command line flags
configPath := flag.String("config", "config.json", "Path to the configuration file")
flag.Parse()
func listFiles(dirPath string) {
logger.Infof("Files in %s:", dirPath)
err := filepath.Walk(dirPath, func(path string, info os.FileInfo, err error) error {
if err != nil {
logger.Errorf("Error accessing file: %s", err)
return nil
}
if !info.IsDir() {
logger.Infof("- %s", path)
}
return nil
// Initialize logrus logger
logger := logrus.New()
logger.SetFormatter(&logrus.TextFormatter{
FullTimestamp: true,
TimestampFormat: "2006-01-02 15:04:05",
})
// Read the configuration file
configFile, err := os.Open(*configPath)
if err != nil {
logger.Errorf("Error listing files: %s", err)
logger.Fatalf("Error opening configuration file: %v", err)
}
defer configFile.Close()
// Decode the configuration file
var config Config
err = json.NewDecoder(configFile).Decode(&config)
if err != nil {
logger.Fatalf("Error decoding configuration file: %v", err)
}
// Start a goroutine to restart the server periodically
if config.Heartbeat > 0 {
go func() {
for range time.Tick(time.Duration(config.Heartbeat)) {
logger.Info("Server heartbeat")
os.Exit(0)
}
}()
}
// Create a new CDNHandler with the configuration
cdnHandler := &CDNHandler{
Config: config,
Logger: logger,
}
// Create a new HTTP server
server := &http.Server{
Addr: ":" + config.Port,
Handler: cdnHandler,
ErrorLog: log.New(logger.Writer(), "", 0),
ReadTimeout: 5 * time.Second,
WriteTimeout: 10 * time.Second,
}
logger.Infof("Starting CDN server on port %s", config.Port)
log.Fatal(server.ListenAndServe())
}