-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Description
Describe the bug
I am using PostObjectV4 to get presigned post credentials, so users can upload videos from their browser directly to my S3 bucket. When I add the data from PostObjectV4 to my HTML form fields, and attempt to upload a test mp4 video (1.5MB in size) using the POST method, I get these errors:
<Error>
<Code>MethodNotAllowed</Code>
<Message>The specified method is not allowed against this resource.</Message>
<RequestId>767327875541C1DB:B</RequestId>
<HostId>P3QUMcxiDLe1ZYkWSpXsRV6A6O200XIpOTMkL0GWfJcM6uF+iWB4HCVJgOGVdcVIVdyAVSHEyfNP</HostId>
<CMReferenceId>MTY5NDQ3ODE5ODg1MSAzOC4yNy4xMDYuMTAxIENvbklEOjE3NDQwNTYwNS9FbmdpbmVDb25JRDoyMjU3MDQxL0NvcmU6Mzc=</CMReferenceId>
</Error>
Chrome Developer Tools Info:
**Request Headers:**
POST /v.mysite.com HTTP/1.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Cache-Control: max-age=0
Connection: keep-alive
Content-Length: 742
Content-Type: application/x-www-form-urlencoded
Host: s3.*********.com
Origin: https://www.mysite.com
Referer: https://www.mysite.com/
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: cross-site
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36
sec-ch-ua: "Chromium";v="116", "Not)A;Brand";v="24", "Google Chrome";v="116"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
**Response Headers:**
HTTP/1.1 405 Method Not Allowed
Access-Control-Allow-Headers: *
Access-Control-Allow-Methods: GET, HEAD, POST, PUT, DELETE, MOVE, OPTIONS
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Date, Etag, Content-Length, Accept-Ranges, Content-Range, Server, Location, X-Amz-Version-Id
Access-Control-Max-Age: 86400
Connection: close
Content-Type: application/xml
Date: Tue, 12 Sep 2023 00:23:19 GMT
Server: *******S3/7.16.1950-2023-08-24-c6d9c0fd32 (B26-U13)
x-amz-id-2: P3QUMcxiDLe1ZYkWSpXsRV6A6O200XIpOTMkL0GWfJcM6uF+iWB4HCVJgOGVdcVIVdyAVSHEyfNP
x-amz-request-id: 767327875541C1DB:B
x-*********-cm-reference-id: 1694478198851 ***.***.***.*** ConID:174405605/EngineConID:2257041/Core:37
Transfer-Encoding: chunked
**Payload:**
Content-Type: application/x-www-form-urlencoded
Policy: eyJleHBpcmF0aW9uIjoiMjAyMy0wOS0xMlQyMDoyMjo1OFoiLCJjb25kaXRpb25zIjpbeyJhY2wiOiJwdWJsaWMtcmVhZCJ9LHsiYnVja2V0Ijoidi5sZXRzaGFuZ291dC5jb20ifSx7IlgtQW16LURhdGUiOiIyMDIzMDkxMlQwMDIyNThaIn0seyJYLUFtei1DcmVkZW50aWFsIjoiQTg2U0I2TktBMUtXQzRFRjhaMjVcLzIwMjMwOTEyXC91cy1lYXN0LTFcL3MzXC9hd3M0X3JlcXVlc3QifSx7IlgtQW16LUFsZ29yaXRobSI6IkFXUzQtSE1BQy1TSEEyNTYifV19
X-Amz-Algorithm: AWS4-HMAC-SHA256
X-Amz-Credential: ***************************/20230912/us-east-1/s3/aws4_request
X-Amz-Date: 20230912T002258Z
X-Amz-Signature: c175f65574afa42a7c9696135429e8c45d84f5393d5810504f8940b2d71416b2
acl: public-read
content-type: video/mp4
key: test-video.mp4
filename: test-video.mp4
I've tweaked all sorts of variables, cut my example down to the simplest example possible, but S3 is just failing constantly. I've set my bucket to complete public access and allowed all actions on the bucket policy for testing purposes - nothing should be blocking upload. I've been doing back and forth with my cloud storage hosting company for a few weeks and we are getting nowhere. Apparently, they cannot reproduce the error, but they are not giving me specific details on what they are seeing either.
My bucket policy:
[{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "*",
"Resource": "arn:aws:s3:::v.mysite.com"
}
]
}]Expected Behavior
PostObjectV4 should give me presigned post credentials and form input values that are good for 20 hours, which I can insert into HTML form fields and the user can select a file on their device to upload. Using the 'POST' method, a user's file should upload directly to the S3 server without errors. After the video is uploaded, it should be immediately available for the public to view in their browser (public-read).
Current Behavior
I receive the PostObjectV4 form fields and add those to my HTML form. The user selects the file to upload, and when it gets sent, there are 405 errors - Method Not Allowed.
Reproduction Steps
The full PHP/HTML code example:
define('AWS_KEY', '********************');
define('AWS_SECRET_KEY', '****************************************');
define('HOST', 'https://s3.**********.com');
define('REGION', 'us-east-1');
require_once("aws-autoloader.php");
use Aws\S3\S3Client;
$cdn = new Aws\S3\S3Client([
'version' => '2006-03-01',
'region' => REGION,
'endpoint' => HOST,
'credentials' => [
'key' => AWS_KEY,
'secret' => AWS_SECRET_KEY,
]
]);
$mykey = 'test-video.mp4';
$bucket = 'v.mysite.com';
$formInputs = [
'acl' => 'public-read',
'key' => $mykey,
'content-type' => 'video/mp4',
];
$options = [
['acl' => 'public-read'],
['bucket' => $bucket]
];
$expires = '+20 hours';
$postObject = new \Aws\S3\PostObjectV4(
$cdnvideo,
$bucket,
$formInputs,
$options,
$expires
);
$formAttributes = $postObject->getFormAttributes();
$formInputs = $postObject->getFormInputs();
?>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body>
<form action="<?php echo $formAttributes['action']; ?>" method="POST" enctype="<?php echo $formAttributes['enctype']; ?>">
<input type="hidden" name="policy" value="<?php echo $formInputs['Policy']; ?>" />
<input type="hidden" name="x-amz-algorithm" value="<?php echo $formInputs['X-Amz-Algorithm']; ?>" />
<input type="hidden" name="x-amz-credential" value="<?php echo $formInputs['X-Amz-Credential']; ?>" />
<input type="hidden" name="x-amz-date" value="<?php echo $formInputs['X-Amz-Date']; ?>" />
<input type="hidden" name="x-amz-signature" value="<?php echo $formInputs['X-Amz-Signature']; ?>" />
<input type="hidden" name="acl" value="public-read" />
<input type="hidden" name="content-type" value="video/mp4" />
<input type="hidden" name="key" value="<?php echo $mykey; ?>" />
<input type="file" name="filename">
<input type="submit">
</form>
</body>
</html>Possible Solution
Something wrong with or related to my Nginx, PHP or SSL configurations???
Additional Information/Context
I enabled total public/admin access to my bucket policy and bucket. All actions are allowed, there should be nothing restricting uploads for my test. My entire PHP/HTML script consists of this, about as simple as it gets:
define('AWS_KEY', '********************');
define('AWS_SECRET_KEY', '****************************************');
define('HOST', 'https://s3.**********.com');
define('REGION', 'us-east-1');
require_once("aws-autoloader.php");
use Aws\S3\S3Client;
$cdn = new Aws\S3\S3Client([
'version' => '2006-03-01',
'region' => REGION,
'endpoint' => HOST,
'credentials' => [
'key' => AWS_KEY,
'secret' => AWS_SECRET_KEY,
]
]);
$mykey = 'test-video.mp4';
$bucket = 'v.mysite.com';
$formInputs = [
'acl' => 'public-read',
'key' => $mykey,
'content-type' => 'video/mp4',
];
$options = [
['acl' => 'public-read'],
['bucket' => $bucket]
];
$expires = '+20 hours';
$postObject = new \Aws\S3\PostObjectV4(
$cdnvideo,
$bucket,
$formInputs,
$options,
$expires
);
$formAttributes = $postObject->getFormAttributes();
$formInputs = $postObject->getFormInputs();
?>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body>
<form action="<?php echo $formAttributes['action']; ?>" method="POST">
<input type="hidden" name="Policy" value="<?php echo $formInputs['Policy']; ?>" />
<input type="hidden" name="X-Amz-Algorithm" value="<?php echo $formInputs['X-Amz-Algorithm']; ?>" />
<input type="hidden" name="X-Amz-Credential" value="<?php echo $formInputs['X-Amz-Credential']; ?>" />
<input type="hidden" name="X-Amz-Date" value="<?php echo $formInputs['X-Amz-Date']; ?>" />
<input type="hidden" name="X-Amz-Signature" value="<?php echo $formInputs['X-Amz-Signature']; ?>" />
<input type="hidden" name="acl" value="public-read" />
<input type="hidden" name="content-type" value="video/mp4" />
<input type="hidden" name="key" value="<?php echo $mykey; ?>" />
<input type="file" name="filename">
<input type="submit">
</form>
</body>
</html>
Important notes:
My bucket is called: v.mysite.com, and it's also a subdomain that I use to host my images on. The subdomain has a CNAME set up and proxied through CloudFlare to: v.mysite.com.s3.[cloudstorage].com. I have a free Universal Edge certificate set up on CloudFlare, which provides SSL to that subdomain as well. So I can host my videos using something like https://v.mysite.com/video.mp4 , it looks nicer and professional.
Edit: Should mention that I had created a separate non-subdomain-name bucket (no CloudFlare proxy) as a test, and that didn't work either...
Also important: that v.mysite.com bucket works fine if users upload the video to my server and I process it and send it to S3 using the putObject method. I've been doing it for years now with images/videos. However, to prevent my server from overloading if many users upload videos at the same time, I would rather have them upload directly from client browser to S3, and the PostObjectV4 strategy doesn't work. I tried using a similar getSignedUrl('putObject') strategy as well with the same failed results.
Here are my nginx.conf file (using nginx version: nginx/1.23.2 compiled with modsecurity):
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
load_module modules/ngx_http_modsecurity_module.so;
events {
use epoll;
worker_connections 768;
multi_accept on;
}
http {
# Expires map (cache control)
map $sent_http_content_type $expires {
default off;
text/html epoch;
text/css 14d;
application/javascript 14d;
~image/ 14d;
~font/ 14d;
access_log off;
}
include /etc/nginx/conf.d/default;
include /etc/nginx/conf.d/*.com;
fastcgi_read_timeout 300s;
index index.php index.htm index.html;
client_max_body_size 10M;
##ModSecurity##
modsecurity on;
modsecurity_rules_file /etc/nginx/modsec/modsec-config.conf;
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
server_tokens off;
add_header Cache-Control "public";
add_header Strict-Transport-Security 'max-age=31536000; includeSubDomains; preload';
add_header X-XSS-Protection "1; mode=block";
add_header X-Frame-Options "SAMEORIGIN";
add_header X-Content-Type-Options nosniff;
add_header Referrer-Policy "strict-origin";
add_header Permissions-Policy "geolocation=(),midi=(),sync-xhr=(),microphone=(),camera=(),magnetometer=(),gyroscope=(),fullscreen=(self),payment=()";
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# SSL Settings
##
#ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
# SSL Notes for A+ SSL Labs score: https://techlabs.blog/categories/how-to-guides/get-an-ssl-labs-a-rating-for-your-nginx-website
ssl_protocols TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
ssl_prefer_server_ciphers off;
ssl_session_cache shared:SSL:10m; # 1m holds approx 4000 sessions
ssl_session_timeout 1d; # 1 hour during which sessions can be re-used
ssl_session_tickets off;
ssl_stapling on;
ssl_stapling_verify on;
resolver 1.1.1.1 1.0.0.1;
##
# Logging Settings
##
access_log /var/log/nginx/access.log combined buffer=16k;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
gzip_vary on;
gzip_proxied any;
gzip_comp_level 5;
gzip_buffers 16 8k;
gzip_http_version 1.1;
gzip_min_length 256;
gzip_types application/atom+xml application/geo+json application/javascript application/x-javascript application/json application/ld+json application/manifest+json application/rdf+xml application/rss+xml application/xhtml+xml application/xml font/eot font/otf font/ttf font/opentype image/bpm image/svg+xml image/x-icon text/css text/javascript text/plain text/xml text/x-component text/x-cross-domain-policy;
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
I disabled modsecurity, restarted nginx and tested it, with no success.
mysite.com.conf Nginx configs:
server {
server_name www.mysite.com;
root /home/username/mysite.com;
charset utf-8;
expires $expires;
listen [::]:443 http2 ssl; # managed by Certbot
listen 443 http2 ssl; # managed by Certbot
rewrite ^([^.]*[^/])$ $1/ permanent;
location / {
try_files $uri $uri/ =404;
}
location ~ \.php$ {
fastcgi_pass unix:/var/run/php/php8.1-fpm-username.sock;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
include fastcgi_params;
}
ssl_certificate /etc/letsencrypt/live/www.mysite.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/www.mysite.com/privkey.pem; # managed by Certbot
}
server {
server_name mysite.com;
return 301 https://www.mysite.com$request_uri;
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/mysite.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/mysite.com/privkey.pem; # managed by Certbot
}
server {
server_name mysite.com www.mysite.com;
listen 80;
return 301 https://www.mysite.com$request_uri;
}
I have attached my PHP.ini file. I am using PHP 8.1 FPM:
PHP 8.1.23 (cli) (built: Sep 2 2023 06:59:15) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.1.23, Copyright (c) Zend Technologies
with Zend OPcache v8.1.23, Copyright (c), by Zend Technologies
Lastly, my cloud storage hosting support asked me to run a cURL function as a test in the shell and provide the output. This is the cURL response:
sudo curl -vvv -F 'acl=public-read' -F 'key=/user/test/original/1694480698e7c1acf3b42c4ev.mp4' -F 'content-type=video/mp4' -F 'X-Amz-Credential=************************/20230912/us-east-1/s3/aws4_request' -F 'X-Amz-Algorithm=AWS4-HMAC-SHA256' -F 'X-Amz-Date=20230912T010458Z' -F 'Policy=eyJleHBpcmF0aW9uIjoiMjAyMy0wOS0xMlQyMTowNDo1OFoiLCJjb25kaXRpb25zIjpbeyJhY2wiOiJwdWJsaWMtcmVhZCJ9LHsiYnVja2V0Ijoidi5sZXRzaGFuZ291dC5jb20ifSx7IlgtQW16LURhdGUiOiIyMDIzMDkxMlQwMTA0NThaIn0seyJYLUFtei1DcmVkZW50aWFsIjoiQTg2U0I2TktBMUtXQzRFRjhaMjVcLzIwMjMwOTEyXC91cy1lYXN0LTFcL3MzXC9hd3M0X3JlcXVlc3QifSx7IlgtQW16LUFsZ29yaXRobSI6IkFXUzQtSE1BQy1TSEEyNTYifV19' -F 'X-Amz-Signature=3339d5ab27e9388ca5392b79cf2f7b33e5e3f5fae03a71beed0b31560765c6ec' -F 'Content-Type=video/mp4' -F 'file=test.mp4' 'https://s3.********.com/v.mysite.com'
* Trying ***.***.***.***:443...
* Connected to s3.********s.com ( ***.***.***.***) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: C=US; ST=Massachusetts; L=Boston; O=***** Technologies LLC; CN=*.s3.*******.com
* start date: Sep 23 00:00:00 2022 GMT
* expire date: Oct 24 23:59:59 2023 GMT
* subjectAltName: host "s3.********.com" matched cert's "s3.********.com"
* issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS RSA SHA256 2020 CA1
* SSL certificate verify ok.
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> POST /v.mysite.com HTTP/1.1
> Host: s3.**********.com
> User-Agent: curl/7.81.0
> Accept: */*
> Content-Length: 1617
> Content-Type: multipart/form-data; boundary=------------------------5a840f371118ebb7
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* We are completely uploaded and fine
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
< Connection: close
< Content-Type: application/xml
< Date: Tue, 12 Sep 2023 01:05:26 GMT
< Server: ***********S3/7.16.1979-2023-09-08-cd1d698b50 (head02)
< x-amz-id-2: hL2xpiJvBwvJkta+8HSZji/9N7PX3QpSoxuMh0W0qQ31UdDGxbGbH/pY2TDLmlxvrLg5D3e+4Ysm
< x-amz-request-id: B7296EC48440E6BE:A
< Transfer-Encoding: chunked
<
* TLSv1.2 (IN), TLS header, Supplemental data (23):
<?xml version="1.0" encoding="UTF-8"?>
* Closing connection 0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS alert, close notify (256):
<Error><Code>AccessDenied</Code><Message>Invalid according to Policy: Policy Condition failed: ["eq", "$X-Amz-Credential", "**************************\/20230912\/us-east-1\/s3\/aws4_request"]</Message><RequestId>B7296EC48440E6BE:A</RequestId><HostId>hL2xpiJvBwvJkta+8HSZji/9N7PX3QpSoxuMh0W0qQ31UdDGxbGbH/pY2TDLmlxvrLg5D3e+4Ysm</HostId></Error>
I was asked to remove the ACL='public-read', and also try ACL='private', but that didn't have any affect. I'm running out of ideas on what is causing this. I'm guessing maybe my Nginx/PHP/SSL configs are messing with S3. I might need to remove/fix/add something in these configs? Or this could be a bug related to the type of setup I'm using.
SDK version used
3.281.4
Environment details (Version of PHP (php -v)? OS name and version, etc.)
PHP 8.1.23 FPM, Ubuntu 22.04, Nginx 1.23.2